Italian translation at settesei.it
There is a persistent belief among tennis fans and commentators that some players are particularly inconsistent. For today’s purposes, I’m talking about match-to-match results, the players who have a knack for upsetting higher-ranked opponents but are also particularly susceptible to losses against weaker players. We have a range of words for this, like unpredictable, dangerous, tricky, and the preferred term for Nick Kyrgios: mercurial.
So far in 2019, Kyrgios has provided a perfect example of the inconsistent type. After early losses to Jeremy Chardy and Radu Albot, he bounced back to win last week’s ATP 500 in Acapulco, knocking out Rafael Nadal, Stan Wawrinka, John Isner, and Alexander Zverev. There’s no question that the Australian possesses more talent than his ranking would suggest. This is a guy who has yet to crack the top ten, but holds a .500 record in completed matches against the Big 3, a feat managed by no other active player (minimum 5 matches, excepting Nadal and Novak Djokovic themselves).
He sounds inconsistent. His results look unpredictable. But compared to the uncertainty that comes with every tennis match between highly-ranked professionals, how does he stack up? As my headline suggests, it’s not as clear-cut as it seems.
Measuring predictability
Consider the opposite type, a player who reliably beats lower-ranked opponents and usually loses against his betters. Roberto Bautista Agut has this type of reputation. As we’ll see, the numbers bear it out, notwithstanding his Doha upset of Djokovic a couple of months ago. If someone really is so predictable, that should show up in a comparison of his pre-match forecasts to his results. For a Bautista Agut type, the forecasts would be particularly accurate, while for a Kyrgios type, the forecasts would be much less reliable.
We already have a metric for this. Brier Score measures the accuracy of forecasts, considering not just how often predictions proved correct, but how close they came. For instance, after Kyrgios beat Zverev in Saturday’s Acapulco final, those prognosticators who gave the Aussie a 90% chance of winning were “more” correct than those who gave him a 60% shot. On the other hand, too much confidence runs the risk of a worse Brier Score–if you’re always giving tennis favorites a 90% chance of winning, you’ll often be wrong. Brier Score is the average of the squared difference between the pre-match forecast (e.g. 90%) and the result (1 or 0, depending if the pick was correct).
Brier Scores for ATP forecasting hover around the 0.2 mark. A lower Brier Score is better, representing less difference between prediction and results, so if you can come in much lower than 0.2, you should be making money betting on matches. If you’re much higher than 0.2, you might as well be flipping a coin. If we use random, 50/50 pre-match predictions, the resulting Brier Score is 0.25.
Brier-gios
If a player is truly unpredictable, the Brier Score for his matches should approach the 0.25 mark, and it should definitely exceed the tour-typical 0.2. To measure the reliability of pre-match forecasts for Kyrgios and other players, I used my surface-weighted Elo ratings for every completed tour-level main draw match since 2000 and generated percentage forecasts for each one. By this method, Zverev had a 67.4% probability of winning the Acapulco final.
So far in 2019, Kyrgios does look truly unpredictable. The Brier Score of his ten match results is 0.318, meaning that we’d have done better by simply flipping a coin to forecast the result of each of his matches. Even if we retroactively increase his chances of winning each match to account for the fact that he’s playing better than his Elo rating predicted, the Brier Score is 0.277, still worse than coin flips.
On the other hand, it’s just ten matches. Several other players have 2019 Brier Scores well over the 0.25 threshold, including Frances Tiafoe, Joao Sousa, Juan Ignacio Londero, and Felix Auger Aliassime. In a handful of tournaments, you’ll always get a few oddball results, either because of marked improvements (as is likely with Auger Aliassime) or extreme good or bad luck. Unless we’re willing to say that Sousa and Londero are remarkably unpredictable players, we shouldn’t draw the same conclusion based on Kyrgios’s last ten matches.
What you predict is what you get
The Brier Score for Elo-based forecasts of Kyrgios’s career matches at tour level is 0.219. That’s higher–and thus less predictable–than average, but not by that much. Of the 280 players with at least 100 tour-level matches this century, Kyrgios ranks 84th, more reliable than 30% of his peers. In 2017, his results were quite unpredictable, with a Brier Score of 0.244, but in 2015 and 2016 they generated a more pedestrian 0.210, and last year they looked downright predictable, at 0.177.
The Australian may be quite unpredictable in tactics, point-to-point performance, or on-court behavior, but his results just aren’t that unusual. The following table shows the 15 most unpredictable active players, as measured by Brier Score, along with Kyrgios, followed by the 15 most predictable active players:
Player Matches Brier Lucas Pouille 189 0.247 Andrey Rublev 106 0.245 Benoit Paire 377 0.239 Ivo Karlovic 650 0.239 Stefanos Tsitsipas 100 0.232 Karen Khachanov 154 0.231 Peter Gojowczyk 102 0.231 Federico Delbonis 225 0.227 Marius Copil 108 0.227 Damir Dzumhur 173 0.227 Ernests Gulbis 420 0.226 Pablo Cuevas 338 0.226 Mischa Zverev 297 0.226 Joao Sousa 323 0.226 Borna Coric 210 0.226 ... Nick Kyrgios 191 0.219 ... Matthew Ebden 171 0.188 David Goffin 344 0.188 Marin Cilic 684 0.186 Richard Gasquet 770 0.183 Tomas Berdych 911 0.182 Milos Raonic 448 0.178 David Ferrer 1048 0.177 Jo Wilfried Tsonga 600 0.175 Roberto Bautista Agut 384 0.172 Kei Nishikori 517 0.167 Juan Martin Del Potro 560 0.160 Andy Murray 802 0.146 Roger Federer 1350 0.121 Novak Djokovic 951 0.117 Rafael Nadal 1060 0.114
Lucas Pouille’s results have been almost impossible to forecast. The Brier Score generated by his 2018 results was nearly 0.3, suggesting it would have been smarter to calculate a forecast and then bet against it! Ivo Karlovic also shows up among the less reliable players, though it’s not clear whether that’s due to his unusual game style. Isner, the only decent parallel we have, is as reliable as the tour in general, with a career Brier Score of 0.201. Reilly Opelka, the other towering ace machine in the ATP top 100, has defied the odds so far in 2019, but he hasn’t yet amassed enough data to draw any conclusions.
At the other end of the spectrum, the most reliable players are many of the best. That adds up: A dominant player not only wins most of the matches he should, but his performance also allows us to make more aggressive forecasts. Nadal often enters matches with a 90% or better probability of winning, and confident predictions like that–as long the player converts them into wins–are what generate the lowest Brier Scores.
Consistent consistency results
We all tend to read too much into unusual results. Kyrgios has given us plenty of those, and we’ve repaid the favor by making him out to be even more of a wild card than he is. A couple of weeks ago, I took on a similar question and found that ATPers don’t really “play their way in” to tournaments, earning better or worse results in different rounds. This isn’t quite the same issue, but it all comes back to similar truths: Existing forecasts are pretty good, there’s always going to be a lot of randomness in the results, and the stories we invent to account for the randomness don’t really explain much at all.
Kyrgios is an immensely interesting player–I joked in yesterday’s podcast that readers should prepare themselves for a ten-part series–and digging into his point-by-point stats could reveal characteristics that are unique among tour players. That is still true. But at the match level, the likelihood that his contests will end in upsets isn’t unique at all–even if he is the proud new owner of a sombrero that says otherwise.