When a calibration “defect” is just noise

A peer check we didn’t expect

padelapi.org runs an independent padel prediction model. They compared their win probabilities against ours across thousands of settled matches. The reassuring part came first: both models are well calibrated, and essentially perfect at the sharp end — when either of us says a heavy favourite wins 82%, favourites in that bucket really do win about 82% of the time.

But both models had the same apparent soft spot, in the 55–60% band, with opposite signs. On our side that band looked underconfident: we predicted around 57%, and the players in it were winning closer to 63%. It looked like a small, fixable flaw in how the model separated near-even matchups.

The tempting fix

The obvious move is to bend the calibration curve until that band sits on the diagonal. It is easy to do and it makes the reliability chart look better instantly. If the gap is real, that is exactly the right fix.

The catch is the word “if.” A reliability diagram splits predictions into bins, and each bin is a small sample with its own random error. A band that looks off might be a genuine discrimination gap — or it might just be the wobble you get from a few hundred coin-flippy matches. Those two look identical on the chart. The only way to tell them apart is to test out of sample.

What the time split showed

We re-ran the whole thing as a strict walk-forward: train the model and its calibration on the earlier 70% of matches by date, then report only on the later 30% that the tuning never touched. On that honest split the story fell apart. The worst-calibrated band was no longer 55–60% — it had moved to 60–65%. And when we pushed the model hard enough to “fix” the middle, the bands did not flatten; they overshot into overconfidence, and the worst band simply relocated somewhere else.

padelapi ran their own out-of-sample check and landed in the same place: their isotonic recalibration, which looked like a clean improvement in sample, swung from −3.4 to +5.1 once tested on held-out data. Two models, two different methods, one verdict: the mid-band was binning noise, not a fixable defect. Bending the curve to chase it would have traded a real, stable model for a chart that looked prettier and predicted worse.

The real improvement, hiding in plain sight

Ruling out the false lead is not a wasted trip — it made room for the true one. While validating on the time split, we found a change that did survive it. Our rating system already weighted margin of victory, but the weight was far too conservative: ratings crawled through the middle of the field, so temperature calibration had to do most of the sharpening.

Giving lopsided wins more say moved the held-out Brier score from 0.202 to 0.188 — a paired improvement of about 3.3 standard errors, well past the bar for chance. Accuracy rose too, and temperature scaling, which had been carrying the calibration, went nearly idle: the ratings now carry the sharpness themselves. Crucially, the held-out log-loss bottoms out exactly at the setting we shipped, so it is neither under- nor over-tuned. That is the difference between the two changes — one survived the split, the other did not.

The playbook

The lesson generalises well beyond padel, to anyone reading a calibration chart:

• Always validate on a time-split holdout. Both of us found in-sample “optima” that did not survive out of sample.
• A reliability diagram is many bins at once, each with its own sampling error. An off-looking band is a hypothesis, not a finding.
• Before bending a curve to meet a band, prove the band is signal. If a small change relocates the problem instead of removing it, it was noise.
• The sharp end (65%+) is where both independent models agree and stay reliable — that is the part you can trust, and it never needed touching.