Every surprising score in gymnastics raises a conversation about domestic scoring. We debate whether Aly Raisman’s beam score was too low at the 2015 World Championships, and whether we should believe that Russia can challenge for a team silver based on their scores at national championships.
But is domestic overscoring really such a big issue? I ran the numbers on all the gymnastic scores collected by 4for4.info and The Gymternet. This dataset includes scores from 151 competitions from the past quad, with 2,524 gymnasts from 94 countries.
Gymnasts get better and worse over time, so it’s important to compare their domestic and international scores over a relatively short period. I chose to compare scores within a single “competition year” – that is, the period between two world championships. For example, the 2015 competition year started after the Tokyo World Championships in October 2014 and ended after the Glasgow World Championships in October 2015.
Gymnasts were only included in the analysis if they competed both domestically and internationally in the same competition year. The average gymnast competing at a domestic meet scores lower than the average gymnast competing at an international meet because only the best get international assignments. So in order to ensure that we compare domestic and international scores for gymnasts of the same caliber, I only looked at those who did both.
First, I calculated each gymnast’s average domestic score and average international score on each event in each competition year. Then, I took the difference between the gymnasts’ domestic and international scores and calculated the average of all these differences for each country. This tells us how much higher the average domestic score is than the average international score in America, Russia, China, and so on.
Here are the results:
A few key takeaways from this graph:
- Most countries don’t inflate their scores domestically. For eight countries – USA, Germany, the Netherlands, France, Russia, Spain, Japan, and Belgium – the difference between domestic and international scores was within the margin of error.
- There are four countries – Austria, Romania, Australia, and Italy – that do have significantly higher scores at domestic meets. Of these, Austria is the worst offender, with gymnasts scoring more than five tenths higher at home. But these four countries aren’t exactly the ones that are most often accused of egregious score inflation.
- Three countries – the UK, Canada, and China – actually appear to deflate their gymnasts’ scores at home. China deflates scores the most, with gymnasts scoring more than two tenths lower at home than they do at international meets.
Surprising, right? But of course, there are dozens of factors that might explain why the results above show so little domestic overscoring – after all, those numbers are just simple means. Maybe lots of gymnasts are coming back from injury at the early season domestic meets, and they peak for international meets later in the season – for example, Aliya Mustafina has done her best work at the European Championships for the past two years. Maybe they don’t bring their full difficulty to domestic means the way they do at international meets – there’s no need for Giulia Steingruber to throw her top skills at the Swiss Championships, and she doesn’t.
That’s why I also created a regression model to estimate domestic overscoring for each country. This model includes a bunch of those other important factors: How early in the season was the meet? How consistent are the gymnast’s scores at domestic meets? Does it matter which apparatus the gymnast is on?
Controlling for all these other factors, how much higher are domestic scores in each country? The results from this model look remarkably similar to the initial results.
This model still shows that Austria, Romania, and Australia inflate domestic scores, while Canada and the UK still deflate domestic scores. And again, most gymnasts from most countries don’t score significantly higher at domestic meets.
But really, what this analysis tells us is that there’s a lot of variation in a gymnast’s scores that can’t be explained by domestic overscoring. In technical terms, domestic judging isn’t a significant predictor of a gymnast’s scores, even controlling for all the factors we can measure. Sure, domestic scores seem a bit too high – but that hardly matters when you take into account who has her best day at the Olympics and who has her worst. When we think about how a gymnast is likely to do in the future, we should focus on her difficulty, her health, her consistency, and her overall readiness for competition. It’s not worth arguing about the accuracy of her scores from the past.
Of course, we can still watch Madison Kocian’s beautiful bars set and know that it won’t get a 15.9 at the Olympics the way it did at the Olympic Trials. But we shouldn’t let individual cases change our understanding of the overall trend. Domestic overscoring isn’t actually such a big deal.
Questions on the methodology? Just e-mail me at email@example.com.