Every surprising score in gymnastics raises a conversation about domestic scoring. We debate whether Aly Raisman’s beam score was too low at the 2015 World Championships, and whether we should believe that Russia can challenge for a team silver based on their scores at national championships.
But is domestic overscoring really such a big issue? I ran the numbers on all the gymnastic scores collected by 4for4.info and The Gymternet. This dataset includes scores from 151 competitions from the past quad, with 2,524 gymnasts from 94 countries.
Gymnasts get better and worse over time, so it’s important to compare their domestic and international scores over a relatively short period. I chose to compare scores within a single “competition year” – that is, the period between two world championships. For example, the 2015 competition year started after the Tokyo World Championships in October 2014 and ended after the Glasgow World Championships in October 2015.
Gymnasts were only included in the analysis if they competed both domestically and internationally in the same competition year. The average gymnast competing at a domestic meet scores lower than the average gymnast competing at an international meet because only the best get international assignments. So in order to ensure that we compare domestic and international scores for gymnasts of the same caliber, I only looked at those who did both.
First, I calculated each gymnast’s average domestic score and average international score on each event in each competition year. Then, I took the difference between the gymnasts’ domestic and international scores and calculated the average of all these differences for each country. This tells us how much higher the average domestic score is than the average international score in America, Russia, China, and so on.
Here are the results:
A few key takeaways from this graph:
- Most countries don’t inflate their scores domestically. For eight countries – USA, Germany, the Netherlands, France, Russia, Spain, Japan, and Belgium – the difference between domestic and international scores was within the margin of error.
- There are four countries – Austria, Romania, Australia, and Italy – that do have significantly higher scores at domestic meets. Of these, Austria is the worst offender, with gymnasts scoring more than five tenths higher at home. But these four countries aren’t exactly the ones that are most often accused of egregious score inflation.
- Three countries – the UK, Canada, and China – actually appear to deflate their gymnasts’ scores at home. China deflates scores the most, with gymnasts scoring more than two tenths lower at home than they do at international meets.
Surprising, right? But of course, there are dozens of factors that might explain why the results above show so little domestic overscoring – after all, those numbers are just simple means. Maybe lots of gymnasts are coming back from injury at the early season domestic meets, and they peak for international meets later in the season – for example, Aliya Mustafina has done her best work at the European Championships for the past two years. Maybe they don’t bring their full difficulty to domestic means the way they do at international meets – there’s no need for Giulia Steingruber to throw her top skills at the Swiss Championships, and she doesn’t.
That’s why I also created a regression model to estimate domestic overscoring for each country. This model includes a bunch of those other important factors: How early in the season was the meet? How consistent are the gymnast’s scores at domestic meets? Does it matter which apparatus the gymnast is on?
Controlling for all these other factors, how much higher are domestic scores in each country? The results from this model look remarkably similar to the initial results.
This model still shows that Austria, Romania, and Australia inflate domestic scores, while Canada and the UK still deflate domestic scores. And again, most gymnasts from most countries don’t score significantly higher at domestic meets.
But really, what this analysis tells us is that there’s a lot of variation in a gymnast’s scores that can’t be explained by domestic overscoring. In technical terms, domestic judging isn’t a significant predictor of a gymnast’s scores, even controlling for all the factors we can measure. Sure, domestic scores seem a bit too high – but that hardly matters when you take into account who has her best day at the Olympics and who has her worst. When we think about how a gymnast is likely to do in the future, we should focus on her difficulty, her health, her consistency, and her overall readiness for competition. It’s not worth arguing about the accuracy of her scores from the past.
Of course, we can still watch Madison Kocian’s beautiful bars set and know that it won’t get a 15.9 at the Olympics the way it did at the Olympic Trials. But we shouldn’t let individual cases change our understanding of the overall trend. Domestic overscoring isn’t actually such a big deal.
Questions on the methodology? Just e-mail me at brina.seidel@gmail.com.
The Gymternet 
Love this!!
LikeLike
Love this, definitely interesting to read!
I assume at the top though, you meant Nanning World Championships, not Tokyo?
LikeLike
I think the truth about overscoring is that location matters as much or more than anything else. First of all I think in general the home team is always overscored. Then you can draw a fairly strong correlation between geopolitical relations (which affect everyday human perception and bias) and scores awarded. Correct me if I’m wrong but the US has only been to one “international” meet outside of the NA/EU zone in the past quad.
LikeLike
Yea, those Europeans sure do love America and American gymnasts.
LikeLike
Yep, the gymnasts and the western values they represent. Beware, I’ve prepared, a simple snide dismissal doesn’t get you much. Please decide if you want to do a numbers to numbers talk on the affect of home countries and alliances.
LikeLike
Ok, so basically you are just saying that USA gymnasts don’t get overscored…Um, no matter how many excel graphs you make, they are still grossly over-scored.
LikeLike
Basically, that’s not what this article says at all.
Enjoy your day!
LikeLike
Didn’t read all of it, just first paragraph.
I assume politics about scoring system in this code I suppose. But honestly whatever score is given, that shit right there on that pic.
It’s picture perfect, that shit dope.
To give an imperfect score would just be hating or agenda based.
Good is good, you gots to give credit to good, regardless of whatever.
LikeLike
Doesn’t fit in, but will you do a Live Blog/Reflection fir Podiums Training? I probably can’t watch it, so that would mean a lot to me 🙂
LikeLike
The Balance Beam Situation did a mostly USA live blog. Of non-USA note, Hong un Jong did a TTY in training and Shang Chunsong fell a lot.
LikeLike
Also Rebeca Andrade performed an Amanar, a 6.5 D-Score Bars Routine, and performed the 1.5 Twisting Double Tuck to get named after her in the quals. Giulia Steingruber upgraded her 3rd pass to Double Layout. And Ellie Downie did a Patterson as beam dismount.
LikeLike
So what’s the motivation for the domestic scoring inflation where it occurs? Is it to excite the home crowd? (I would think that the scoring is so complicated, that people don’t understand it enough for that to actually happen). Or to pressure the international judging panel? Or that the judges just aren’t that well trained? Or that the judges know they are being generous but fear there will be consequences if they aren’t?
I’m also interested in whether most of the discrepancy comes from the E score. Do they just take a more “glass is half full” perspective at home? I would think the D score is less subjective.
LikeLike
Lauren, off-topic but have you heard anything about Paseka? Wasn’t she supposed to have an evaluation yesterday?
LikeLike
Nvm just saw a video of her podium training I guess she’s cleared.
LikeLike
http://deadspin.com/all-306-olympic-medal-events-ranked-1784673580
I know this has nothing to do with this article, but can we just….I know it’s Deadspin, but this dude is wrong about soooo many things in this article. Balance beam is not Simone’s best event! Aly Raisman will not have trouble making the floor final in 2016! 2016 is not necessarily Simone’s last Olympics! Paseka was hardly the favorite to win the vault title before she hurt her back!
At least he ranked the majority of the gymnastics events in the Top 50, but Jesus, do some research, dude.
LikeLike
SO ANNOYING! Seriously why do all the mainstream sports writers just look at world championship results and assume that’s what is going to happen at the Olympics??? This isn’t swimming or track. Everything changes in gymnastics day by day. All the talk about Simone and vaulting and how that’s the one medal she might not win like GIVE ME A BREAK PEOPLE! And the BB situation is even worse. Like Simone lost to Laurie 3/4 times this summer on beam. She is inconsistent on it. If she hits, she will win, but she’s not a lock to hit.
LikeLike
Yeah but Laurie was SO overscored domestically. I think Simone is slightly better with both hit routines, and she’s much more experienced. And, although neither of them looked perfect, Simone’s first routine vs Laurie’s… second routine? (I can’t remember which was better), Simone def would’ve scored higher. I mean, it really just depends.
LikeLike
I still think it’s Simone’s third-best event. To ignore her floor routine, which has moved in it that are named for her, not to mention height and tumbling that men struggle to do well, is just bad research and laziness all around.
LikeLike
I just saw the same article and left a scolding comment about Aly having trouble making the floor final. In what universe? Maybe the author only looked at the 2015 world championships for research?
Of course, if sports media actually put any effort into covering women’s sports maybe this kind of thing wouldn’t happen. There’s no way a sports writer would get away with making such basic mistakes about football, baseball or men’s basketball!
LikeLike
Holy god, that was so bad. I really just roasted them in the comments. Honestly, don’t even write about gymnastics if you know that little. I love how they don’t even mention Laurie hernandez and I think they called the team final the, “Team All Around” Like are you combining the Team Final and the All Around final into one?
LikeLike
Actually, the official Olympics name for the team competition is “Team All-Around,” as contrasted with “Individual All-Around.” It’s kind of an awkward name, and it’s not referred to that way in other gym competitions, but he wasn’t actually wrong about that part.
LikeLike
General question for anyone- Is there a way to watch the podium training of other, non-US athletes? I know the US livestream starts at 4:30, but I would love to see some of the other countries, as well! The same question also goes for qualifications, because I’m assuming NBC will not show us a very diverse group 🙂
LikeLike
Wow just saw Lauren’s links to the other streams in meet coverage, too bad I didn’t think to look earlier 😦
LikeLike
Best I have found so far, lots of videos uploaded from different countries gymnasts here!
https://www.instagram.com/gymnastics_planet/
LikeLike
Thanks!
LikeLike
Also, do you know if the NBC livestream is just for the USA’s subdivision or will they be showing Russia/China/Brazil/Germany/etc.
LikeLike
I do think NBC will broadcast all subdivisions live on the website and they won’t just show the US. The stream is the official one going to all TV stations around the world and NBC has no say in what is shown. They’re later broadcast of the competition, however, will most likely be focused on USA. If you’re on the US and have cable, NBC is probably a good option. If not, BBC has everything for every sport and is the one I’ll be watching. You do need a VPN set to UK, though. Hope this helps
LikeLike
It seems from an observation standpoint, not a statistical one, that it is particular individuals at domestic meets that get overscored rather than all gymnasts getting overscored in a domestic meet. For example, as GORGEOUS as kocian’s bar routine was at trials, 15.9 seemed really high. Same with Simone’s 16.2 amanar and laurie’s 15.7 beam. This is not to say their routines weren’t fantastic, because they all were, they just seemed particularly high. However, scores for Maggie Nichols, Reagan smith and others seemed as though they were not overscored. Just an observation.
LikeLike
By the lineup, it seems like Laurie isn’t doing all around. Which is very disappointing. But it’s also contradicting to what Marta said earlier about lineups. She said anybody going up on that event in team finals will go up in qualifying. And by the lineup it looks like Laurie isn’t doing bars in qualifying. But they will probably use her in team finals.
LikeLike
That’s what I was thinking too, except Laurie looked ROUGH on bars. That plus her being a first year senior would make me a little nervous about using her in team finals and not quals! I’m hoping it was just nerves (because even Simone seemed nervous) because I really want Laurie to be in the bars lineup in the team final!
LikeLike
Same. But Laurie is gonna have to go up on bars in team final because there is no way they are using Simone or Aly in team final
LikeLike
Nah Simone will go up on bars in team finals. She will lead off
LikeLike
Sorry, I also have a not-related-to-overscoring just general Olympics question: on the link that was posted in the mega You Asked regarding NBC’s Olympic coverage, it mentioned that there will be concurrent streams for each apparatus. Will the concurrent streams be on nbcolympics.com, or do you need the app for that?
LikeLike
This must have been a ton of work to compile. I’d be curious to see how individual gymnasts’ domestic scores compare to international. Because I think some of the problem with U.S. scoring is not overscoring so much as partisan scoring. They overscore the favorites and underscore those who are not among the chosen. So if you average those, it looks like the scoring is not that much different, but it doesn’t mean the domestic scores accurately reflect the international ones.
LikeLike
I can’t help but come back and laugh (good-naturedly) at this article’s ending note, heh.
“Of course, we can still watch Madison Kocian’s beautiful bars set and know that it won’t get a 15.9 at the Olympics the way it did at the Olympic Trials.”
Weeeeeeelllll….. we might’ve been wrong about that.
LikeLike
Yeah, the Olympic bars scores were insane as a whole…I wasn’t expecting ANYONE to score what they did!
LikeLike