{"id":2129,"date":"2017-01-15T16:06:50","date_gmt":"2017-01-15T16:06:50","guid":{"rendered":"http:\/\/www.tennisabstract.com\/blog\/?p=2129"},"modified":"2017-01-15T16:06:50","modified_gmt":"2017-01-15T16:06:50","slug":"measuring-the-performance-of-tennis-prediction-models","status":"publish","type":"post","link":"https:\/\/www.tennisabstract.com\/blog\/2017\/01\/15\/measuring-the-performance-of-tennis-prediction-models\/","title":{"rendered":"Measuring the Performance of Tennis Prediction Models"},"content":{"rendered":"

With the recent buzz about Elo rankings in tennis, both at FiveThirtyEight<\/a> and here at Tennis Abstract<\/a>, comes the ability to forecast the results of tennis matches. It’s not far fetched to ask yourself, which of these different models perform better and, even more interesting, how they fare compared to\u00a0other ‘models’, such as the ATP ranking system or betting markets.<\/p>\n

For this, admittedly limited, investigation, we collected the (implied)\u00a0forecasts of five models, that is, FiveThirtyEight<\/a>, Tennis Abstract, Riles<\/a>, the official ATP rankings, and the Pinnacle betting market<\/a>\u00a0for the US Open 2016. The first three\u00a0models are based on Elo.\u00a0For inferring forecasts from the ATP ranking, we use a specific formula1<\/sup><\/a> and for Pinnacle, which is one of the biggest tennis bookmakers, we calculate the implied probabilities based on the provided odds (minus the overround<\/a>)2<\/sup><\/a>.<\/p>\n

Next, we simply compare forecasts with reality for each model asking\u00a0If player A was predicted to be the\u00a0winner ($latex P(a) > 0.5$), did he really win the match?\u00a0<\/em>When we do that for each match and\u00a0each model (ignoring retirements or walkovers) we come up with the following results.<\/p>\n

Model\t\t% correct\nPinnacle\t76.92%\n538\t\t75.21%\nTA\t\t74.36%\nATP\t\t72.65%\nRiles\t\t70.09%\n<\/pre>\n
What we see here is how many percent of the predictions were actually right. The betting model (based on the odds of Pinnacle) comes out on top followed by the Elo models of FiveThirtyEight and Tennis Abstract. Interestingly, the Elo model of Riles is outperformed by the predictions inferred from the ATP ranking. Since there are several parameters that can be used to tweak an Elo model, Riles may still have some room left for improvement.<\/p>\n
However, just looking at the percentage of correctly called matches does not tell the whole story. In fact, there are more granular metrics to investigate the performance of a prediction model:\u00a0Calibration<\/em>, for instance, captures the ability of a model to provide forecast probabilities that are close to the true probabilities. In other words, in an ideal model, we want 70% forecasts to be true exactly in 70% of the cases.\u00a0Resolution<\/em> measures how much the forecasts differ from the overall average. The rationale here is, that just using the expected average values for forecasting will lead to a reasonably well-calibrated set of predictions, however, it will not be as useful\u00a0as a method that\u00a0manages the same calibration while taking current circumstances\u00a0into account. In other words, the more extreme (and still correct) forecasts are, the better.<\/p>\n
In the following table we categorize\u00a0the set of predictions into bins of different probabilities and show how many percent of the predictions were correct per bin. This also enables us to\u00a0calculate\u00a0Calibration<\/em> and\u00a0Resolution<\/em> measures for each model.<\/p>\n
Model    50-59%  60-69%  70-79%  80-89%  90-100% Cal  Res   Brier\n538      53%     61%     85%     80%     91%     .003 .082  .171\nTA       56%     75%     78%     74%     90%     .003 .072  .182\nRiles    56%     86%<\/strong>     81%     63%     67%     .017 .056  .211\nATP      50%     73%     77%     84%     100%    .003 .068  .185\nPinnacle 52%     91%<\/strong>     71%     77%     95%     .015 .093  .172\n<\/pre>\n
As we can see, the predictions are not always perfectly in line with what the corresponding bin would suggest. Some of these deviations, for instance the fact that for the Riles model only 67% of the 90-100% forecasts were correct, can be explained by small sample size (only three\u00a0in that case). However, there are still two interesting cases (marked in bold) where sample size is better and which raised my interest. Both the Riles and Pinnacle models seem to be strongly underconfident<\/em>\u00a0(statistically significant) with their 60-69% predictions. In other words, these probabilities should have been higher, because, in reality, these forecasts were actually true 86% and 91% percent of the times.3<\/sup><\/a> For the betting aficionados, the fact that Pinnacle underestimates the favorites here may be really interesting, because it could reveal some\u00a0value<\/em> as punters would say. For the Riles model, this would maybe be a starting point to tweak the model.<\/p>\n
In the last three columns\u00a0Calibration<\/em> (the lower the better),\u00a0Resolution <\/em>(the higher the better), and the\u00a0Brier score\u00a0<\/em>(the lower the better) are shown. The Brier score<\/a> combines Calibration and Resolution (and the uncertainty of the outcomes) into a single score for measuring the accuracy of predictions. The models of FiveThirtyEight and Pinnacle (for the used subset of data) essentially perform equally good.\u00a0Then there is a slight gap until the model of Tennis Abstract and the ATP ranking model come in third and fourth, respectively. The Riles model performs worst in terms of both Calibration and Resolution, hence, ranking fifth in this analysis.<\/p>\n
To conclude, I would like to show a common visual representation that is used to graphically display a set of predictions. The\u00a0reliability diagram compares the observed rate of forecasts with the forecast probability (similar to the above table).<\/p>\n
<\/a><\/p>\n
The closer one of the colored lines is to the black line, the more reliable the forecasts are. If the forecast lines are above the black line, it means that forecasts are\u00a0underconfident<\/em>, in the opposite case, forecasts are\u00a0overconfident<\/em>. Given that we only investigated one tournament and therefore had to work with a low sample size (117 predictions), the big swings in the graph are somewhat expected. Still, we can see that the model based on ATP rankings does a really good job in preventing overestimations even though it is known to be outperformed by Elo in terms of prediction accuracy.<\/p>\n
To sum up, this analysis shows how different predictive models for tennis can be compared among each other in a meaningful way. Moreover, I hope I could exhibit some of the areas where a model is good and where it’s bad. Obviously, this investigation could go into\u00a0much more detail\u00a0by, for example, comparing the models in how well they do for different kinds of players (e.g., based on ranking), different surfaces, etc. This is something I will spare for later. For now, I’ll try to get my sleeping patterns accustomed to the schedule of play for the Australian Open, and I hope,\u00a0you can do the same.<\/p>\n
—<\/p>\n
Peter Wetz<\/a> is a computer scientist interested in racket sports and data analytics based in Vienna, Austria.<\/p>\n
Footnotes<\/h4>\n
1.<\/span> $latex P(a) = a^e \/ (a^e + b^e) $ where $latex a $ are player A’s ranking points, $latex b $ are player B’s ranking points, and $latex e $ is a constant. We use $latex e = 0.85 $ for ATP men’s singles.<\/p>\n
2.<\/span> The betting market in itself is not really a model, that is, the goal of the bookmakers is simply to balance their book. This means that the odds, more or less, reflect the wisdom of the crowd<\/a>, making it a very good predictor.<\/p>\n
3.<\/span> As an example, one instance, where Pinnacle was underconfident and all other models were more confident is the R32 encounter between Ivo Karlovic<\/a> and Jared Donaldson<\/a>. Pinnacle’s implied probability for Karlovic\u00a0to win was 64%. The other models (except the also underconfident Riles model) gave 72% (ATP ranking), 75% (FiveThirtyEight), and 82% (Tennis\u00a0Abstract). Turns out, Karlovic\u00a0won in straight sets. One factor at play here might be that these were the US Open where more US citizens are likely to be confident about the US player Jared Donaldson and hence place a bet on him. As a consequence, to balance the book, Pinnacle will lower the odds on Donaldson, which results in higher odds (and a lower implied probability) for Karlovic.<\/p>\n","protected":false},"excerpt":{"rendered":"
With the recent buzz about Elo rankings in tennis, both at FiveThirtyEight and here at Tennis Abstract, comes the ability to forecast the results of tennis matches. It’s not far fetched to ask yourself, which of these different models perform better and, even more interesting, how they fare compared to\u00a0other ‘models’, such as the ATP … Continue reading Measuring the Performance of Tennis Prediction Models<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[32,40,45,47,91],"tags":[],"class_list":["post-2129","post","type-post","status-publish","format-standard","hentry","category-elo-ratings","category-forecasting","category-grand-slams","category-guest-posts","category-rankings"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/posts\/2129","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/comments?post=2129"}],"version-history":[{"count":0,"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/posts\/2129\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/media?parent=2129"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/categories?post=2129"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tennisabstract.com\/blog\/wp-json\/wp\/v2\/tags?post=2129"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}