Model evaluation
Model vs market
There is one question any football model has to answer before it deserves attention: is it better than the forecast implied by the market? The closing line, the market consensus immediately before kick-off, is the reference standard in sports forecasting, and we measured ourselves against it. We publish no odds and no betting advice: the market appears here purely as a yardstick.
Over 810 matches across 6 seasons (2020-21 to 2025-26): overall and early in the season, the market is ahead; in the rest of the season, it is a statistical tie.
We write this with the error bars in plain sight: a gap smaller than twice its standard error is consistent with a tie, and we read it as one. The current model was picked among variants backtested on these same seasons, so this evaluation is not independent of that choice and tends to favour it (this matters mostly for the two statistical ties, not the overall gap). The first clean test will be the 2026-27 season, whose forecasts are not yet scored here.
The result, in three numbers
RPS difference between the model and the closing line, match by match. Negative = the model errs less. 810 matches, 6 seasons.
Every matchday evaluated
Games from matchdays 2–11, 15, 19, 23, 27 and 31
+0.0050
standard error 0.0016 · n = 810
The market is ahead: the gap is more than twice its standard error (t = 3.07).
Early season
Games from matchdays 2–11
+0.0059
standard error 0.0020 · n = 540
The market is ahead: the gap is more than twice its standard error (t = 2.99).
Rest of the season
Games from matchdays 15, 19, 23, 27 and 31
+0.0031
standard error 0.0028 · n = 270
A statistical tie: the sign favours the market, but the gap is under twice its standard error (t = 1.10).
The gap through the season
Forecast error (RPS) for the model and the closing line at each matchday forecast. Lower is better. The gap between model and market changes sign three times: between matchdays 2 and 3, between matchdays 19 and 23 and between matchdays 23 and 27.
Model and market forecast error, matchday by matchday
Vertical axis truncated (0.14 to 0.21) so the differences are visible; the real differences are the ones in the lower strip. Each point pools one matchday's games across the 6 seasons (n = 54 matches per point), forecast with the results up to the previous matchday. In the lower strip, a bar whose ±2 standard-error line crosses zero is a statistical tie. With 15 matchdays, one or two are expected to pass 2 standard errors by chance alone: the main reading is the three cards', which pool every match.
View as table: Model and market RPS by matchday forecast
| Matchday forecast | Model | Market | Difference | Std. error | t | Matches |
|---|---|---|---|---|---|---|
| 2 | 0.1615 | 0.1635 | −0.0020 | 0.0071 | −0.28 | 54 |
| 3 | 0.1654 | 0.1588 | +0.0067 | 0.0062 | 1.08 | 54 |
| 4 | 0.2035 | 0.1971 | +0.0065 | 0.0058 | 1.11 | 54 |
| 5 | 0.1812 | 0.1716 | +0.0096 | 0.0065 | 1.49 | 54 |
| 6 | 0.1477 | 0.1465 | +0.0013 | 0.0064 | 0.20 | 54 |
| 7 | 0.1915 | 0.1872 | +0.0043 | 0.0058 | 0.73 | 54 |
| 8 | 0.1700 | 0.1644 | +0.0055 | 0.0063 | 0.87 | 54 |
| 9 | 0.1740 | 0.1639 | +0.0101 | 0.0054 | 1.86 | 54 |
| 10 | 0.1784 | 0.1764 | +0.0020 | 0.0060 | 0.33 | 54 |
| 11 | 0.1882 | 0.1727 | +0.0155 | 0.0072 | 2.16 | 54 |
| 15 | 0.1786 | 0.1749 | +0.0038 | 0.0058 | 0.65 | 54 |
| 19 | 0.1834 | 0.1830 | +0.0004 | 0.0057 | 0.07 | 54 |
| 23 | 0.1622 | 0.1666 | −0.0044 | 0.0060 | −0.74 | 54 |
| 27 | 0.1848 | 0.1808 | +0.0039 | 0.0067 | 0.58 | 54 |
| 31 | 0.1922 | 0.1802 | +0.0120 | 0.0075 | 1.60 | 54 |
How to read this
- RPS: lower is better
- The Ranked Probability Score measures how far a win-draw-loss forecast lands from what happened, penalising big misses more. Zero is a perfect forecast; putting 33% on each result scores about 0.23 to 0.24, given the Liga's share of draws.
- What the closing line is
- It is the market's last price before kick-off, with the margin stripped out (Shin's method). By then it has absorbed everything: injuries, line-ups, suspensions, and the money of people who know more than we do.
- Why it is hard to beat
- The closing line is the consensus of thousands of bettors with money at stake, and it is the benchmark any model is measured against. In the same sample, the closing line errs 0.0026 less than the opening line (t = −3.09): the market sharpens up to kick-off. Against the opening line, the model is at +0.0024 (t = 1.70): it is a statistical tie.
A note on uncertainty, because here it decides everything: we read a gap as real only when it is more than twice its standard error; below that, it is a statistical tie.
When we disagreed with the market
Model and market almost always see the same match. These are the cases where they did not.
In 108 of the 810 matches (13.3%) model and market named different favourites. In those, the market's pick won 41 times and the model's 25, with 42 landing on a third result. The error gap on that subset is +0.0172, with a standard error of 0.0069 (t = 2.50): the market is ahead. It is a subset of 108 matches, picked by the predictions rather than the results; the main answer is the whole sample's, above.
- Famalicão 1–1 Rio Ave2020-21 · matchday 3NeitherModel:Famalicão51%Market:Rio Ave43%
- Famalicão 0–1 Gil Vicente2020-21 · matchday 11Market rightModel:Famalicão53%Market:Gil Vicente37%
- Santa Clara 2–1 Sp. Braga2025-26 · matchday 31Market rightModel:Sp. Braga52%Market:Santa Clara38%
- Farense 3–3 Famalicão2020-21 · matchday 4NeitherModel:Famalicão46%Market:Farense42%
- Gil Vicente 1–0 Portimonense2020-21 · matchday 2Model rightModel:Gil Vicente47%Market:Portimonense40%
- Belenenses SAD 1–1 Moreirense2021-22 · matchday 4NeitherModel:Belenenses SAD43%Market:Moreirense44%
- Moreirense 0–1 Paços de Ferreira2020-21 · matchday 7Market rightModel:Moreirense37%Market:Paços de Ferreira50%
- Vizela 2–2 Gil Vicente2022-23 · matchday 4NeitherModel:Gil Vicente48%Market:Vizela39%
The 8 matches where the two favourites were furthest apart. “Neither” means the third result came in.
Leaders win the title more often than the model says
This page evaluates one thing: the win, draw or loss forecast for each game of the next matchday. Title and relegation probabilities are a different question, and we check them separately.
Read with care: in past seasons the model underrated teams leading by 3 or more points (it gave them 63% on average, and they won the title 82% of the time, leaving out two cases with a data error).
We checked what happens to the teams in front. Across the 2017-18 to 2025-26 seasons there were 19 evaluated moments (in 8 seasons) when a team led by 3 or more points. Two are from matchday one and come from a data error (rounds in which the same team appears twice), so we leave them out. In the other 17, the model gave the leader, on average, a 63% chance of winning the title; the leader won it 82% of the time. With the two faulty cases, the figures would be 56% and 74%.
In other words: the model tends to underrate the leader. The sample is small: 17 moments, which are not independent (the same season and the same leader count several times), so the effective sample is smaller still and the exact size of the gap is uncertain, but the direction holds. Nor do we know how to fix it yet: none of the alternatives tested passed our checks, so the published model has not changed. Some of those alternatives, which the checks cannot tell apart from the current model, would have given Porto between 57% and 62% in the 2026-27 matchday 7 forecast, instead of 51%.
Read the title probabilities as a probably conservative estimate for whoever is in front. Relegation probabilities have not been checked this way. (Internal assessment of 25 September 2026.)
Calibration of the points ranges
Over 9 past seasons, from 2017-18 to 2025-26 (162 team-seasons, each checked at 6 points in the season: 972 forecasts, which are not independent of one another), the 90% interval for final points contained the real total 89.3% of the time.
The small print
Evaluated on 810 matches: 6 seasons (2020-21 to 2025-26) × 15 matchdays (games from matchdays 2–11, 15, 19, 23, 27 and 31). To forecast each matchday, the model is fitted only on the matches played up to the one before, without seeing the future. Market probabilities are derived from closing odds published by football-data.co.uk (Pinnacle for 773 matches, Bet365 for 37), with the margin removed using Shin's method. Evaluation generated on 5 October 2026. The model evaluated is the current model (bivariate Poisson with shots on target), the one producing the forecasts published on this site.