Men’s Doubles On the Dirt

Angelique Kerber wasn’t the only top seed to crash out early at this year’s French Open. In the men’s doubles draw, the top section opened up when Henri Kontinen and John Peers, the world’s top-ranked team, lost to the Spanish pair of David Marrero and Tommy Robredo. It’s plausible to attribute the upset to the clay, as Kontinen-Peers have tallied a pedestrian five wins against four losses on the dirt this season and one could guess that the Spaniards are at their strongest on clay.

Fortunately we don’t have to guess. Using a doubles variant of sElo–surface-specific Elo, which I began writing about a few days ago in the context of women’s singles–we can make rough estimates of how Kontinen/Peers would fare against Marrero/Robredo on each surface. The top seeds are solid on all surfaces–less than a year ago, they won a clay title in Hamburg–but stronger on hard courts. sElo ranks them 4th and 8th on hard, but 10th and 13th on clay among tour regulars.  Marrero is the surface-specialist of the bunch, ranking 37th on clay and 78th on hard. Robredo throws a wrench into the exercise, as he has played very little doubles recently, only eight events since the beginning of 2016.

Using these numbers–including those derived from Robredo’s limited sample–we find that sElo would have given Kontinen/Peers a 73.6% chance of winning yesterday, compared to a 78.3% advantage on a hard court. Even if we adjust Robredo’s clay-court sElo to something closer to his all-surface rating, the top seeds still look like 69% favorites.

A more striking example comes from yesterday’s other big upset, in which Julio Peralta and Horacio Zeballos took out Feliciano Lopez and Marc Lopez. On any surface, the Lopezes are the superior team, but Peralta and Zeballos have a much larger surface differential:

Player    Hard sElo  Clay sElo  
M Lopez        1720       1804  
F Lopez        1713       1772  
Zeballos       1651       1756  
Peralta        1517       1770

On a hard court, sElo gives the Lopezes a 68.1% chance of winning this matchup. But on clay, the gap narrows all the way to 53.6%. It’s still a bit of an upset for the South Americans, but not one that should come as much of a surprise.


I’ve speculated in the past that surface preferences aren’t as pronounced in doubles as they are in singles. Regardless of surface, points are shorter, and many teams position one player at the net even on the dirt. While some hard-courters are probably uncomfortable on clay (and vice versa), I wouldn’t expect the effects to be as substantial as they are in singles.

The numbers tell a different story. Here are the top ten, ranked by hard court sElo:

Rank  Player          Hard sElo  
1     Jack Sock            1947  
2     Nicolas Mahut        1893  
3     Marcelo Melo         1883  
4     Henri Kontinen       1879  
5     P-H Herbert          1862  
6     Bob Bryan            1851  
7     Mike Bryan           1846  
8     John Peers           1842  
9     Bruno Soares         1829  
10    Jamie Murray         1828

By clay court sElo:

Rank  Player                Clay sElo  
1     Mike Bryan                 1950  
2     Bob Bryan                  1950  
3     P-H Herbert                1894  
4     Nicolas Mahut              1889  
5     Jack Sock                  1887  
6     Robert Farah               1850  
7     Juan Sebastian Cabal       1849  
8     Pablo Cuevas               1824  
9     Rohan Bopanna              1812  
10    John Peers                 1810

Jamie Murray and Bruno Soares, who appear in the hard court top ten, sit outside the top 25 in clay court sElo. Robert Farah and Juan Sebastian Cabal are 41st and 42nd in hard court sElo, despite ranking in the clay court top seven. Pablo Cuevas, another clay court top-tenner, is 87th on the hard court list.

To go beyond these anecdotes–noteworthy as they are–we need to compare the level of surface preference in men’s doubles to other tours. To do that, I calculated the correlation coefficent between hard court and clay court sElo for the top 50 players (ranked by overall Elo) in men’s doubles, men’s singles, and women’s singles. (I don’t yet have an adequate database to generate ratings for women’s doubles.)

In other words, we’re testing how much a player’s results on one surface predict his or her results on the other major surface. The higher the correlation coefficient, the more likely it is that a player will have similar results on hard and clay. Here’s how the tours compare:

Tour             Correl  
Men's Singles     0.708  
Women's Singles   0.417  
Men's Doubles     0.323

In contrast to my hypothesis above, surface preferences in men’s doubles appear to be much stronger than in either men’s or women’s singles. (And there’s a huge difference between men’s and women’s singles, but that’s a subject for another day.)


I suspect that the low correlation of surface-specific Elos in men’s doubles is partly due to the more random nature of doubles results. Because the event is more serve-dominated, there are more close sets ending in tiebreaks, and because of the no-ad, super-tiebreak format used outside of Slams, tight matches are decided by a smaller number of points. Thus, every doubles player’s results–and their various Elo ratings–reflect the influence of chance more than the singles results are.

Another consideration–one that I haven’t yet made sense of–is that surface-specific ratings don’t improve doubles forecasts they way that they do men’s and women’s singles predictions. As I wrote on Sunday, sElo represents a big improvement over surface-neutral Elo for women’s forecasts, and in an upcoming post, I’ll be able to make some similar observations for the men’s game. Using Brier score, a measure of the calibration of predictions, we can see the effect of using surface-specific Elo ratings in 2016 tour-level matches:

Tour             Elo Brier  sElo Brier  
Men's Singles        0.202       0.169  
Women's Singles      0.220       0.179  
Men's Doubles        0.171       0.181

The lower the Brier score, the more accurate the forecasts. This isn’t a fluke of 2016: The differences in men’s doubles Brier scores are around 0.01 for each of the last 15 seasons. By this measure, Elo does a very good job predicting the outcome of men’s doubles matches, but the surface-specific sElo represents a small step back. It could be that the smaller sample–using only one surface’s worth of results–is more damaging to forecasts in doubles than it is in singles.

Doubles analytics is particularly uncharted territory, and there’s plenty of work remaining for researchers even in this narrow subtopic. There’s lots of work to do for the world’s top doubles players as well, now that we can point to a noticeably weaker surface for so many of them.

Bouncing Back From a Marathon Third Set

In this year’s edition of the French Open, we’ve already seen two women’s matches charge past the 6-6 mark in the third set. On Sunday, Madison Brengle outlasted Julia Goerges 13-11 in the decider, and yesterday, Kristina Mladenovic overcame Jennifer Brady 9-7 in the final set. Marathon three-setters aren’t as gut-busting as the five-set equivalent on the men’s tour, yet they still require players to go beyond the usual limit of a tour match.

Do marathon three-setters affect the fortunes of those players that move on to the next round? Back in 2012, I published a study showing that men who win marathon five-setters (that is, matches that go to 8-6 or longer) win fewer than 30% of their following matches, a rate far worse than what we would expect, given the quality of their next opponents. It seems likely that long three-setters wouldn’t have the same effect, especially since many top women are willing to play five-setters themselves.

The numbers bear out the intuition. From 2001 to the 2017 Australian Open, there have been 185 marathon three-setters in Grand Slam main draws, and the winners of those matches have gone on to win 42.2% of their next contests. That’s more than the equivalent number for men, and it’s even better than it sounds.

Players who need to go deep into a third set to vanquish an early-round opponent are, on average, weaker than those who win in straight sets, so many of the marathon women would already be considered underdogs in their next matches. Using sElo–surface-specific Elo, which I recently introduced–we see that these 185 marathon women would have been expected to win only 44.0% of their following matches. There may be a real effect here, but it is a minor one, especially compared to the fortunes of players who struggle through marathon five-setters.

I ran the same algorithm for women’s Slam matches that ended at 7-6, 7-5, and 6-4 or 6-3 in the final set. Since only the US Open uses the third-set tiebreak format, the available sample for that score is limited, which may explain a slightly wacky result. For the other scores, we see numbers that are roughly similar to the marathon findings. Winners tend to be underdogs against their next opponents, but there is little, if any, hangover effect:

3rd Set Score  Sample  Next W%  Next ExpW%  
Marathons         185    42.2%       44.0%  
7-6                56    48.2%       42.2%  
7-5               232    43.1%       42.7%  
6-4 / 6-3         421    41.6%       43.2%

In short: A long match often tells us something about the winner’s chances against her next foe, but it’s something that we already knew. The tight three-setter itself–marathon or otherwise–has little effect on her chances later on. That’s good news for Mladenovic, who will be back on court tomorrow against Sara Errani, an opponent likely to give her another grueling workout.

Podcast Episode 9: Roland Garros Preview and the Value of Surface-Specific Forecasts

In the Episode 9 of the Tennis Abstract Podcast, Carl Bialik and I preview the upcoming fortnight at Roland Garros. We discuss possible threats to Rafael Nadal, likely beneficiaries of some wide-open sections of the draw, and a number of lesser-known names worth watching in Paris.

We apologize for the sound quality this week–due to personal commitments, we had to improvise a bit, and it was either a podcast with subpar sound quality or no podcast at all. I think it’s still very listenable, but if you’re sensitive to that sort of thing, you may disagree. In any case, thanks for listening!

Click to listen, subscribe on iTunes, find us on Stitcher, or use our feed to get updates on your favorite podcast software.

The Steadily Less Predictable WTA

Update: The numbers in this post summarizing the effectiveness of sElo are much too high–a bug in my code led to calculating effectiveness with post-match ratings instead of pre-match ratings. The parts of the post that don’t have to do with sElo are unaffected and–I hope–remain of interest.

One of the talking points throughout the 2017 WTA season has been the unpredictability of the field. With the absence of Serena Williams, Victoria Azarenka, and until recently, Petra Kvitova and Maria Sharapova, there is a dearth of consistently dominant players. Many of the top remaining players have been unsteady as well, due to some combination of injury (Simona Halep), extreme surface preferences (Johanna Konta), and good old-fashioned regression to the mean (Angelique Kerber).

No top seed has yet won a title at the Premier level or above so far this year. Last week, Stephanie Kovalchik went into more detail, quantifying how seeds have failed to meet expectations and suggesting that the official WTA ranking system–the algorithm that determines which players get those seeds–has failed.

There are plenty of problems with the WTA ranking system, especially if you expect it to have predictive value–that is, if you want it to properly reflect the performance level of players right now. Kovalchik is correct that the rankings have done a particularly poor job this year identifying the best players. However, there’s something else going on: According to much more accurate algorithms, the WTA is more chaotic than it has been for decades.

Picking winners

Let’s start with a really basic measurement: picking winners. Through Rome, there had been more than 1100 completed WTA matches. The higher-ranked player won 62.4% of those. Since 1990, the ranking system has picked the winner of 67.9% of matches, and topped 70% during several years in the 1990s. It never fell below 66% until 2014, and this year’s 62.4% is the worst in the 28-year time frame under consideration.

Elo does a little better. It rates players by the quality of their opponents, meaning that draw luck is taken out of the equation, and does a better job of estimating the ability level of players like Serena and Sharapova, who for various reasons have missed long stretches of time. Since 1990, Elo has picked the winner of 68.6% of matches, falling to an all-time low of 63.1% so far in 2017.

For a big improvement, we need surface-specific Elo (sElo). An effective surface-based system isn’t as complicated as I expected it to be. By generating separate rankings for each surface (using only matches on that surface), sElo has correctly predicted the winner of 76.2% of matches since 1990, almost cracking 80% back in 1992. Even sElo is baffled by 2017, falling to it’s lowest point of 71.0% in 2017.

(sElo for all three major surfaces is now shown on the Tennis Abstract Elo ratings report.)

This graph shows how effectively the three algorithms picked winners. It’s clear that sElo is far better, and the graph also shows that some external factor is driving the predictability of results, affecting the accuracy of all three systems to a similar degree:

Brier scores

We see a similar effect if we use a more sophisticated method to rate the WTA ranking system against Elo and sElo. The Brier score of a collection of predictions measures not only how accurate they are, but also how well calibrated they are–that is, a player forecast to win a matchup 90% of the time really does win nine out of ten, not six out of ten, and vice versa. Brier scores average the square of the difference between each prediction and its corresponding result. Because it uses the square, very bad predictions (for instance, that a player has a 95% chance of winning a match she ended up losing) far outweigh more pedestrian ones (like a player with a 95% chance going on to win).

In 2017 so far, the official WTA ranking system has a Brier score of .237, compared to Elo of .226 and sElo of .187. Lower is better, since we want a system that minimizes the difference between predictions and actual outcomes. All three numbers are the highest of any season since 1990. The corresponding averages over that time span are .207 (WTA), .202 (Elo), and .164 (sElo).

As with the simpler method of counting correct predictions, we see that Elo is a bit better than the official ranking, and both of the surface-agnostic methods are crushed by sElo, even though the surface-specific method uses considerably less data. (For instance, the clay-specific Elo ignores hard and grass court results entirely.) And just like the results of picking winners, we see that the differences in Brier scores of the three methods are fairly consistent, meaning that some other factor is causing the year-to-year differences:

The takeaway

The WTA ranking system has plenty of issues, but its unusually bad performance this year isn’t due to any quirk in the algorithm. Elo and sElo are structured completely differently–the only thing they have in common with the official system is that they use WTA match results–and they show the same trends in both of the above metrics.

One factor affecting the last two years of forecasting accuracy is the absence of players like Serena, Sharapova, and Azarenka. If those three played full schedules and won at their usual clip, there would be quite a few more correct predictions for all three systems, and perhaps there would be fewer big upsets from the players who have tried to replace them at the top of the game.

But that isn’t the whole story. A bunch of no-brainer predictions don’t affect Brier score very much, and the presence of heavily-favored players also make it more likely that massively surprising results occur, such as Serena’s loss to Madison Brengle, or Sharapova’s ouster at the hands of Eugenie Bouchard. Many unexpected results are completely independent of the top ten, like Marketa Vondrousova’s recent title in Biel.

While some of the year-to-year differences in the graphs above are simply noise, the last several years looks much more like a meaningful trend. It could be that we are seeing a large-scale changing of a guard, with young players (and their low rankings) regularly upsetting established stars, while the biggest names in the sport are spending more time on the sidelines. Upsets may also be somewhat contagious: When one 19-year-old aspirant sees a peer beating top-tenners, she may be more confident that she can do the same.

Whatever influences have given us the WTA’s current state of unpredictability, we can see that it’s not just a mirage created by a flawed ranking system. Upsets are more common now than at any other point in recent memory, whichever algorithm you use to pick your favorites.

Podcast Episode 8: Zverev’s Title, Emerging WTA Favorites, and a New Match Format

In the Episode 8 of the Tennis Abstract Podcast, Carl Bialik and I survey the men’s and women’s fields in Rome and consider what last week’s top-tier events have to tell us about Roland Garros. We touch on Alexander Zverev’s maiden Masters title, the mixed signals of Dominic Thiem’s and Novak Djokovic’s tournaments, the rise of Elina Svitolina, and the continued relevance of Venus Williams.

We also have even more to say about wild cards (not Sharapova’s, I promise!) and dive into the potential of the best-of-five, first-to-four-games format set to debut at the NextGen ATP event in November.

Thanks for listening!

Click to listen, subscribe on iTunes, find us on Stitcher, or use our feed to get updates on your favorite podcast software.

Podcast Episode 7: Champion Simona, King Rafa, and Memories of Pico

In the Episode 7 of the Tennis Abstract Podcast, Carl Bialik and I cover a lot of ground, from Simona Halep’s Madrid title and Kristina Mladenovic’s recent outspokenness, to Rafael Nadal’s unbeaten streak and Dominic Thiem’s rising status as a clay-court contender, along with the inevitability that someone born in the 1990s will eventually win a big ATP title.

Thanks for listening!

Click to listen, subscribe on iTunes, find us on Stitcher, or use our feed to get updates on your favorite podcast software.

Dominic Thiem played Davis Cup in Barcelona. Sort of…

This is a guest post by Peter Wetz.

Last week Dominic Thiem fought his way into the finals of the Barcelona Open by winning against Kyle Edmund, Daniel Evans, Yuichi Sugita, and Andy Murray. Three of these four players play for the same flag and Thiem won against each of them. Thiem is not exactly a champion of the current Davis Cup format–he has opted out of playing for Austria several times and has a rather poor record of 2-3 when he does compete–but in Barcelona he has, at least, shown that he can beat several players from the same country over a short amount of time. And that’s what Davis Cup is about, right?

In this post my goal is to put this statistical hiccup into some context. It is not the first time the Austrian defeated three players of the same nationality at one event: In 2016 at Buenos Aires Thiem already beat three players from Spain. However, given that Spanish players appear much more frequently in draws than Britons do, I will take a closer look.

Since 1990, there have only been three tournaments where a single player faced three players from Great Britain. And only one of these players who faced three Britons won each encounter. The following table shows the three tournaments and each of the matches where a player from Great Britain was faced by the same player. Wally Masur is the only player since 1990 who defeated three players from Great Britain in a single tournament. Thiem remains the only player who achieved this in a tournament outside of the island.

Tournament     Round Winner        Loser           Score
'93 Manchester R32   Wally Masur   Ross Matheson   6-4 6-4
'93 Manchester R16   Wally Masur   Chris Wilkinson 6-3 6-7(4) 6-3
'93 Manchester QF    Wally Masur   Jeremy Bates    6-4 6-3

'97 Nottingham R32   Karol Kucera  Martin Lee      6-1 6-1
'97 Nottingham SF    Karol Kucera  Tim Henman      6-4 2-6 6-4
'97 Nottingham F     Greg Rusedski Karol Kucera    6-4 7-5

'01 Nottingham R32   Martin Lee    Lee Childs      6-4 5-7 6-0
'01 Nottingham R16   Martin Lee    Arvind Parmar   6-4 6-3
'01 Nottingham QF    Greg Rusedski Martin Lee      6-3 6-2

Obviously, there are not many chances to face three Britons in a single tournament. And when one of those opponents is likely to be Andy Murray, a player’s chances of beating all three are even slimmer.

Let’s broaden the perspective a bit and take a look at how often a player defeated three (or more) players from the same country without looking only at Great Britain. The following table displays the results of this analysis. The first column contains the country, the second column (3W) shows how often a player defeated three players of this country, the third column (3WL) shows how often a player defeated two players of this country and then lost to a player of the same country, and so on.

Country  3W  3WL  4W  4WL  5W  5WL
USA      119 179  19  30   1   4
ESP      98  157  17  18   3   2
FRA      28  45   5   2    1   0
ARG      22  26   5   3    0   0
GER      15  18   1   1    0   0
AUS      13  9    0   0    0   0
SWE      9   16   1   0    0   0
CZE      4   5    0   0    0   0
NED      4   4    0   0    0   0
RUS      4   3    0   0    0   0
ITA      2   3    1   0    0   0
BRA      1   3    1   0    0   0
GBR      1   2    0   0    0   0
CHI      1   1    0   0    0   0
SUI      1   1    0   0    0   0

As we could have imagined, USA, ESP, and FRA come out on top here, simply, because for years they have had the highest density of players in the rankings. These are also the only countries of which a player was faced five times at a single tournament. Facing a player of the same country six or more times never happened according to the data at hand. The following table shows the most recent occasions of the entries printed in bold in the above table (5W).

Tournament    Round Winner        Loser             Score
'91 Charlotte R32   Jaime Yzaga   Chris Garner      7-6 6-3
'91 Charlotte R16   Jaime Yzaga   Jimmy Brown       6-4 6-4
'91 Charlotte QF    Jaime Yzaga   Michael Chang     7-6 6-1
'91 Charlotte SF    Jaime Yzaga   M. Washington     7-5 6-2
'91 Charlotte F     Jaime Yzaga   Jimmy Arias       6-3 7-5
'07 Lyon      R32   Sebastien Gr. Rodolphe Cadart   6-3 6-2
'07 Lyon      R16   Sebastien Gr. Fabrice Santoro   4-6 6-1 6-2
'07 Lyon      QF    Sebastien Gr. Julien Benneteau  6-7 6-2 7-6
'07 Lyon      SF    Sebastien Gr. Jo Tsonga         6-1 6-2
'07 Lyon      F     Sebastien Gr. Marc Gicquel      7-6 6-4
'08 Valencia  R32   David Ferrer  Ivan Navarro      6-3 6-4
'08 Valencia  R16   David Ferrer  Pablo Andujar     6-3 6-4
'08 Valencia  QF    David Ferrer  Fernando Verdasco 6-3 1-6 7-5
'08 Valencia  SF    David Ferrer  Tommy Robredo     2-6 6-2 6-3
'08 Valencia  F     David Ferrer  Nicolas Almagro   4-6 6-2 7-6

Finally, we take a look at the big four. Did they ever eliminate three or more players from the same country in a single tournament? Yes, they did. In 2014 Roger Federer beat three Czech players in Dubai. In 2005, 2008, and 2013 he beat three German players in Halle. In 2009 Andy Murray beat three Spanish players in Valencia. In 2007 Novak Djokovic beat three Spanish players in Estoril. In 2013 Rafael Nadal beat three Argentinian players both in Acapulco and Sao Paolo. In 2015 he even beat four Argentinian players in Buenos Aires. And there are many other examples where Rafa beat three of his countrymen at the same tournament.

We can see that this happens fairly often, specifically for countries where the tournament is organized, because more players of this country appear in the draw due to wild cards and qualifications. If we exclude these cases, Federer’s streak in Dubai stands out, as does Thiem’s streak in Barcelona.

Peter Wetz is a computer scientist interested in racket sports and data analytics based in Vienna, Austria.