The Truth About Domestic Overscoring

madison-kocian-kocian_44a

Every surprising score in gymnastics raises a conversation about domestic scoring. We debate whether Aly Raisman’s beam score was too low at the 2015 World Championships, and whether we should believe that Russia can challenge for a team silver based on their scores at national championships.

But is domestic overscoring really such a big issue? I ran the numbers on all the gymnastic scores collected by 4for4.info and The Gymternet. This dataset includes scores from 151 competitions from the past quad, with 2,524 gymnasts from 94 countries.

Gymnasts get better and worse over time, so it’s important to compare their domestic and international scores over a relatively short period. I chose to compare scores within a single “competition year” – that is, the period between two world championships. For example, the 2015 competition year started after the Tokyo World Championships in October 2014 and ended after the Glasgow World Championships in October 2015.

Gymnasts were only included in the analysis if they competed both domestically and internationally in the same competition year. The average gymnast competing at a domestic meet scores lower than the average gymnast competing at an international meet because only the best get international assignments.  So in order to ensure that we compare domestic and international scores for gymnasts of the same caliber, I only looked at those who did both. 

First, I calculated each gymnast’s average domestic score and average international score on each event in each competition year. Then, I took the difference between the gymnasts’ domestic and international scores and calculated the average of all these differences for each country. This tells us how much higher the average domestic score is than the average international score in America, Russia, China, and so on.

Here are the results:

National Overscoring Means

A few key takeaways from this graph:

  1. Most countries don’t inflate their scores domestically. For eight countries – USA, Germany, the Netherlands, France, Russia, Spain, Japan, and Belgium – the difference between domestic and international scores was within the margin of error.
  2. There are four countries – Austria, Romania, Australia, and Italy – that do have significantly higher scores at domestic meets. Of these, Austria is the worst offender, with gymnasts scoring more than five tenths higher at home. But these four countries aren’t exactly the ones that are most often accused of egregious score inflation.
  3. Three countries – the UK, Canada, and China – actually appear to deflate their gymnasts’ scores at home. China deflates scores the most, with gymnasts scoring more than two tenths lower at home than they do at international meets.

Surprising, right? But of course, there are dozens of factors that might explain why the results above show so little domestic overscoring – after all, those numbers are just simple means. Maybe lots of gymnasts are coming back from injury at the early season domestic meets, and they peak for international meets later in the season – for example, Aliya Mustafina has done her best work at the European Championships for the past two years. Maybe they don’t bring their full difficulty to domestic means the way they do at international meets – there’s no need for Giulia Steingruber to throw her top skills at the Swiss Championships, and she doesn’t.

That’s why I also created a regression model to estimate domestic overscoring for each country. This model includes a bunch of those other important factors: How early in the season was the meet? How consistent are the gymnast’s scores at domestic meets? Does it matter which apparatus the gymnast is on?

Controlling for all these other factors, how much higher are domestic scores in each country? The results from this model look remarkably similar to the initial results.

National Overscoring Regression Model

This model still shows that Austria, Romania, and Australia inflate domestic scores, while Canada and the UK still deflate domestic scores. And again, most gymnasts from most countries don’t score significantly higher at domestic meets.

But really, what this analysis tells us is that there’s a lot of variation in a gymnast’s scores that can’t be explained by domestic overscoring. In technical terms, domestic judging isn’t a significant predictor of a gymnast’s scores, even controlling for all the factors we can measure. Sure, domestic scores seem a bit too high – but that hardly matters when you take into account who has her best day at the Olympics and who has her worst. When we think about how a gymnast is likely to do in the future, we should focus on her difficulty, her health, her consistency, and her overall readiness for competition. It’s not worth arguing about the accuracy of her scores from the past. 

Of course, we can still watch Madison Kocian’s beautiful bars set and know that it won’t get a 15.9 at the Olympics the way it did at the Olympic Trials. But we shouldn’t let individual cases change our understanding of the overall trend. Domestic overscoring isn’t actually such a big deal. 

Questions on the methodology? Just e-mail me at brina.seidel@gmail.com.

Advertisements

36 thoughts on “The Truth About Domestic Overscoring

  1. I think the truth about overscoring is that location matters as much or more than anything else. First of all I think in general the home team is always overscored. Then you can draw a fairly strong correlation between geopolitical relations (which affect everyday human perception and bias) and scores awarded. Correct me if I’m wrong but the US has only been to one “international” meet outside of the NA/EU zone in the past quad.

    Like

      • Yep, the gymnasts and the western values they represent. Beware, I’ve prepared, a simple snide dismissal doesn’t get you much. Please decide if you want to do a numbers to numbers talk on the affect of home countries and alliances.

        Like

  2. Ok, so basically you are just saying that USA gymnasts don’t get overscored…Um, no matter how many excel graphs you make, they are still grossly over-scored.

    Like

  3. Didn’t read all of it, just first paragraph.
    I assume politics about scoring system in this code I suppose. But honestly whatever score is given, that shit right there on that pic.
    It’s picture perfect, that shit dope.
    To give an imperfect score would just be hating or agenda based.

    Good is good, you gots to give credit to good, regardless of whatever.

    Like

  4. Doesn’t fit in, but will you do a Live Blog/Reflection fir Podiums Training? I probably can’t watch it, so that would mean a lot to me 🙂

    Like

    • The Balance Beam Situation did a mostly USA live blog. Of non-USA note, Hong un Jong did a TTY in training and Shang Chunsong fell a lot.

      Like

    • Also Rebeca Andrade performed an Amanar, a 6.5 D-Score Bars Routine, and performed the 1.5 Twisting Double Tuck to get named after her in the quals. Giulia Steingruber upgraded her 3rd pass to Double Layout. And Ellie Downie did a Patterson as beam dismount.

      Like

  5. So what’s the motivation for the domestic scoring inflation where it occurs? Is it to excite the home crowd? (I would think that the scoring is so complicated, that people don’t understand it enough for that to actually happen). Or to pressure the international judging panel? Or that the judges just aren’t that well trained? Or that the judges know they are being generous but fear there will be consequences if they aren’t?

    I’m also interested in whether most of the discrepancy comes from the E score. Do they just take a more “glass is half full” perspective at home? I would think the D score is less subjective.

    Like

  6. http://deadspin.com/all-306-olympic-medal-events-ranked-1784673580

    I know this has nothing to do with this article, but can we just….I know it’s Deadspin, but this dude is wrong about soooo many things in this article. Balance beam is not Simone’s best event! Aly Raisman will not have trouble making the floor final in 2016! 2016 is not necessarily Simone’s last Olympics! Paseka was hardly the favorite to win the vault title before she hurt her back!

    At least he ranked the majority of the gymnastics events in the Top 50, but Jesus, do some research, dude.

    Like

    • SO ANNOYING! Seriously why do all the mainstream sports writers just look at world championship results and assume that’s what is going to happen at the Olympics??? This isn’t swimming or track. Everything changes in gymnastics day by day. All the talk about Simone and vaulting and how that’s the one medal she might not win like GIVE ME A BREAK PEOPLE! And the BB situation is even worse. Like Simone lost to Laurie 3/4 times this summer on beam. She is inconsistent on it. If she hits, she will win, but she’s not a lock to hit.

      Like

      • Yeah but Laurie was SO overscored domestically. I think Simone is slightly better with both hit routines, and she’s much more experienced. And, although neither of them looked perfect, Simone’s first routine vs Laurie’s… second routine? (I can’t remember which was better), Simone def would’ve scored higher. I mean, it really just depends.

        Like

        • I still think it’s Simone’s third-best event. To ignore her floor routine, which has moved in it that are named for her, not to mention height and tumbling that men struggle to do well, is just bad research and laziness all around.

          Like

    • I just saw the same article and left a scolding comment about Aly having trouble making the floor final. In what universe? Maybe the author only looked at the 2015 world championships for research?

      Of course, if sports media actually put any effort into covering women’s sports maybe this kind of thing wouldn’t happen. There’s no way a sports writer would get away with making such basic mistakes about football, baseball or men’s basketball!

      Like

    • Holy god, that was so bad. I really just roasted them in the comments. Honestly, don’t even write about gymnastics if you know that little. I love how they don’t even mention Laurie hernandez and I think they called the team final the, “Team All Around” Like are you combining the Team Final and the All Around final into one?

      Like

      • Actually, the official Olympics name for the team competition is “Team All-Around,” as contrasted with “Individual All-Around.” It’s kind of an awkward name, and it’s not referred to that way in other gym competitions, but he wasn’t actually wrong about that part.

        Like

  7. General question for anyone- Is there a way to watch the podium training of other, non-US athletes? I know the US livestream starts at 4:30, but I would love to see some of the other countries, as well! The same question also goes for qualifications, because I’m assuming NBC will not show us a very diverse group 🙂

    Like

    • Also, do you know if the NBC livestream is just for the USA’s subdivision or will they be showing Russia/China/Brazil/Germany/etc.

      Like

    • I do think NBC will broadcast all subdivisions live on the website and they won’t just show the US. The stream is the official one going to all TV stations around the world and NBC has no say in what is shown. They’re later broadcast of the competition, however, will most likely be focused on USA. If you’re on the US and have cable, NBC is probably a good option. If not, BBC has everything for every sport and is the one I’ll be watching. You do need a VPN set to UK, though. Hope this helps

      Like

  8. It seems from an observation standpoint, not a statistical one, that it is particular individuals at domestic meets that get overscored rather than all gymnasts getting overscored in a domestic meet. For example, as GORGEOUS as kocian’s bar routine was at trials, 15.9 seemed really high. Same with Simone’s 16.2 amanar and laurie’s 15.7 beam. This is not to say their routines weren’t fantastic, because they all were, they just seemed particularly high. However, scores for Maggie Nichols, Reagan smith and others seemed as though they were not overscored. Just an observation.

    Like

  9. By the lineup, it seems like Laurie isn’t doing all around. Which is very disappointing. But it’s also contradicting to what Marta said earlier about lineups. She said anybody going up on that event in team finals will go up in qualifying. And by the lineup it looks like Laurie isn’t doing bars in qualifying. But they will probably use her in team finals.

    Like

    • That’s what I was thinking too, except Laurie looked ROUGH on bars. That plus her being a first year senior would make me a little nervous about using her in team finals and not quals! I’m hoping it was just nerves (because even Simone seemed nervous) because I really want Laurie to be in the bars lineup in the team final!

      Like

  10. Sorry, I also have a not-related-to-overscoring just general Olympics question: on the link that was posted in the mega You Asked regarding NBC’s Olympic coverage, it mentioned that there will be concurrent streams for each apparatus. Will the concurrent streams be on nbcolympics.com, or do you need the app for that?

    Like

  11. This must have been a ton of work to compile. I’d be curious to see how individual gymnasts’ domestic scores compare to international. Because I think some of the problem with U.S. scoring is not overscoring so much as partisan scoring. They overscore the favorites and underscore those who are not among the chosen. So if you average those, it looks like the scoring is not that much different, but it doesn’t mean the domestic scores accurately reflect the international ones.

    Like

  12. I can’t help but come back and laugh (good-naturedly) at this article’s ending note, heh.

    “Of course, we can still watch Madison Kocian’s beautiful bars set and know that it won’t get a 15.9 at the Olympics the way it did at the Olympic Trials.”

    Weeeeeeelllll….. we might’ve been wrong about that.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s