“Consistency” is one of the many terms that commentators frequently use but rarely define. It’s often misused, too: we say we want a player to be more consistent, when we really just want him to stop playing badly.
To me, consistency for a tennis player is similar to the notion of “playing up to his ranking.” In other words, if a player is consistent, he usually beats players ranked lower, and he usually loses to players ranked higher. No player is perfect in this regard, but clearly, some are much more reliable than others.
A recent poster boy for inconsistency is Ernests Gulbis. At Roland Garros, he lost to Blaz Kavcic, ranked 82nd in the world. That was on clay, a surface on which Gulbis had posted some excellent results the previous year. Two months later, in Los Angeles, he beat Juan Martin del Potro, someone he shouldn’t have even challenged on a hard court.
Quantifying consistency
With my player rankings and match prediction system, I’m able to assign a win probability to each player for every match. For instance, when Ivan Dodig beat Nadal last week, I had given him a 14.4% chance of doing so. As you might imagine, that’s a major upset–as I wrote the next day, it was the 10th-biggest upset of the season.
In these terms, an ideally consistent player will never be on either end of an upset. If he is the favorite, he wins; if he is the underdog, he loses. In practice, no tour-level player accomplishes this, though over the last two years, Florent Serra and Eduardo Schwank have come very close.
I’ve come up with a metric to measure consistency. This is how it works:
- Gather a list of all ATP-level matches for the desired time period. (Today, I’m using everything from January 2010 through Montreal last week.)
- Eliminate matches that ended in retirement or walkover, as well as those where we don’t have enough information to make an educated prediction. (e.g. the first few comeback matches of Tommy Haas, or one with a wildcard playing his first professional match.)
- For each player, count how many matches he played.
- For each player, find the matches where he was the favorite and lost, or was the underdog and won.
- For each of those matches, take the probability than the eventual winner would win (e.g. 18% — always under 50%), multiply by 100 (e.g. 18, not 18%), subtract it from 50 (e.g. 50 – 18 = 32), and square the result (e.g. 32*32 = 1024).
- Sum all of the squares, then divide by the number of total matches–not just the ones where the favorite lost.
Whew! In something more like layman’s terms, we’re taking all the upsets a player was involved in, coming up with a number to represent how big (or surprising) the upset was, then averaging the results.
Using this method, we give big upsets considerably more weight than mini-upsets. If a player had a 45% chance of winning a match and ends up winning, it barely counts as an upset–and this system treats it accordingly. By dividing by the total number of matches, we give consistency credit to players who win the matches they’re “supposed to” win, and lose those they are supposed to lose.
Most importantly, the numbers this algorithm spits out are completely believable, matching up well with the conventional wisdom of which players are consistent and inconsistent.
The consistency of the top ten
The most consistent player on the tour, since the beginning of 2010, has been … Florent Serra. Amazingly, Igor Kunitsyn comes in second. But I doubt many of you care much about the consistency of guys like that.
Let’s start with the current top 10, ranked from most to least consistent:
Player Upsets Matches Up% UpsetScore David Ferrer 25 119 21.0% 55 Rafael Nadal 16 131 12.2% 68 Novak Djokovic 18 119 15.1% 69 Roger Federer 21 123 17.1% 69 Jo-Wilfried Tsonga 20 80 25.0% 75 Mardy Fish 19 77 24.7% 82 Tomas Berdych 32 113 28.3% 106 Gael Monfils 24 89 27.0% 107 Robin Soderling 23 115 20.0% 130 Andy Murray 24 97 24.7% 151
The relevant column is the rightmost, “UpsetScore,” which is the result of the algorithm described above. Ferrer has been part of more upsets than any of the top three (“Up%”), but his upsets are more minor. Except for losses to Ivo Karlovic and Jarkko Nieminen early in the year on hard courts, Ferrer has not lost a match he had a 60% or better chance of winning.
The two ends of this list certainly line up with what I would have expected: Ferrer and Nadal are rock-solid (last week’s loss to Dodig notwithstanding), while Soderling and Murray both can be picked off by anybody, and frequently threaten higher-ranked players.
Right now, you may be tempted to put Djokovic higher on the list–after all, he’s ranked #1 and he’s beating everybody. However, in the slightly longer term of 20 months, his movement around the top three has included some unexpected results, like losing to Ljubicic at Indian Wells last year, and victories over Federer and Nadal before his ranking suggested he would do so.
Tour wild cards
Outside of the top 10, there are a handful of players who are almost impossible to predict. Some names that come to mind are Marcos Baghdatis, Ernests Gulbis, and Nikolay Davydenko, men who can take out one of the top three on a good day (well, maybe not Gulbis), but can lose to a qualifier on the next.
Player Upsets Matches Up% UpsetScore Nikolay Davydenko 32 73 44% 273 Marin Cilic 27 91 30% 180 Marcos Baghdatis 38 89 43% 177 Olivier Rochus 20 52 38% 164 Milos Raonic 16 41 39% 164 Juan Martin del Potro 11 48 23% 154 Andy Murray 24 97 25% 151 Jurgen Melzer 28 96 29% 150 Fernando Verdasco 40 104 38% 150 Ivan Ljubicic 27 69 39% 149 Florian Mayer 30 72 42% 147 Samuel Querrey 26 66 39% 146 Andrei Goloubev 24 55 44% 143 Ernests Gulbis 22 69 32% 140 Jeremy Chardy 31 65 48% 133 Juan Monaco 26 73 36% 131 Robin Soderling 23 115 20% 130 Michael Llodra 26 67 39% 130 Rainer Schuettler 15 42 36% 119 Mikhail Youzhny 25 78 32% 116
The “upset score” number tells the story for Davydenko. The man who beat Nadal at the beginning of the year and threatened Djokovic last week recently suffered defeat at the hands of Cedrik-Marcel Stebe (twice!) and Antonio Veic.
While no one is in Davydenko’s league, names like Cilic, Baghdatis, Murray, and Verdasco seem appropriate. Verdasco, along with Melzer and Milos Raonic suggest a flaw in this approach: the algorithm reads very fast improvement or decline as inconsistency, which isn’t quite right. Yes, Raonic has shocked the tennis world repeatedly this season, but he hasn’t mixed in too many disastrous losses alongside the surprise upsets. I tinkered with ways to include that in the model, but nothing worked very well.
A couple more interesting notes from the “most inconsistent” players are found in the upset percentage column. Guys like Davydenko, Baghdatis, Mayer, Goloubev, and Chardy are involved in upsets nearly half the time. Chardy is highest in that category. In fact, if I expanded the study to challenger events, he might rocket to the top of this list, as he plays quite a few, and often manages to lose against players outside the top 100.
The consistent ones
The flip side is considerably less star-studded. In the 20 most-consistent players of the last 19-20 months, Ferrer is the only top-10 guy present, though #11 Nicolas Almagro is there as well.
Here’s my seat-of-the-pants theory. In this sense, “consistent” isn’t good. Yes, “consistent” sounds good, especially when “inconsistent” means Davydenko losing to Antonio Veic or Mayer falling to Federico del Bonis. But inconsistent means Davy beating Federer and Mayer beating Soderling. So, the players who show up on as “most consistent” are in fact consistent, but they are also mediocre. Their consistency (perhaps a mental advantage) has helped them move up from the top 200 to the top 50 or 100, but that’s all they can do.
Ferrer and Almagro are good examples of this, actually. Neither has the weaponry that makes commentators say, “This guy could be number one!” But they’ve earned their rankings by regularly reaching the quarters and semis of tournaments, not suffering the boneheaded losses that afflict the likes of Cilic and Baghdatis.
All that said, here’s the list:
Player Upsets Matches Up% UpsetScore Florent Serra 11 56 20% 23 Igor Kunitsyn 14 40 35% 33 Ilia Marchenko 14 46 30% 40 Potito Starace 28 81 35% 46 Victor Hanescu 26 77 34% 50 Tobias Kamke 12 41 29% 52 Andreas Seppi 24 81 30% 53 Julien Benneteau 23 59 39% 53 Viktor Troicki 25 101 25% 54 David Ferrer 25 119 21% 55 Fabio Fognini 18 71 25% 55 Pere Riba 13 41 32% 56 Lukas Lacko 14 44 32% 57 Igor Andreev 17 62 27% 58 Lukasz Kubot 26 63 41% 59 Nicolas Almagro 22 112 20% 59 Frederico Gil 15 40 38% 60 Denis Istomin 25 76 33% 65 Jarkko Nieminen 25 74 34% 66 John Isner 21 82 26% 67
These lists hardly represent the final word on who is or is not consistent–for one thing, I haven’t said anything about consistency within matches, which may be a completely separate issue. But this approach does, I think, provide some insight into who is more likely to be part of an upset, and suggests that consistency might not be such a good thing after all.