The cost of certainty: why Elo can't tell a rookie from a veteran

Two players rated 1500 are not the same player. One number hides that.

The previous page treated every rating as an exact measurement. It isn't. A rating is an estimate, and an estimate should carry an error bar. Elo has none — so it moves a brand-new player and a ten-year veteran as if both were equally known. That is the second structural flaw, and it hits Convocados the moment a Season starts with fresh members.

1 · The missing error bar

Glicko's fix (Glickman, 1995) is to give each player a second number: therating deviation (RD), the system's uncertainty about their rating. A rating is then reported as an interval — a 95% confidence range ofr ± 2·RD.

The update itself respects RD: the bigger your uncertainty, the further one game can move you. This is why a provisional player belongs at the top of a ladder by the end of a night — the system is learning fast because it knows little.

2 · Uncertainty that breathes

Two refinements matter for a weekly hobby league:

• RD grows during inactivity. Glicko widens the interval for players who sit out a rating period, on the assumption their skill may have drifted. A rating frozen for a year is stale evidence.
• Volatility (σ) tracks erratic runs. Glicko-2 adds a third parameter: repeated results that defy the model raise volatility, so the system expects — and permits — bigger swings rather than treating each upset as noise.

3 · Many people, one team

Elo and Glicko are head-to-head. Convocados is 5-a-side: one result, ten players, uneven contribution. Two families handle this:

• TrueSkill models each player's skill as a Gaussian (mean and variance) and infers individual skill from team outcomes via Bayesian inference; it also models draws explicitly.
• OpenSkill is an open, faster implementation of the Weng–Lin Bayesian approximation, built for asymmetric multi-team games and now the practical choice for teams.

Both beat Elo at team settings, but neither knows who was on the bench or how minutes were shared — a caveat Convocados must handle in product, not maths. Team rating is a task for the balancing layer, not necessarily for the motivational ladder.

Check yourself

Primary sources

Work throughGlickman's worked Glicko-2 example— it is short and shows every parameter. For teams, skim theTrueSkill paper, then read theOpenSkill paper for what you'd actually ship.