Forecast Calibration Explained

Forecast calibration is the match between your confidence and reality: you are well calibrated when the things you call 70% likely happen about 70% of the time, the things you call 90% likely happen about 90% of the time, and so on across many predictions.

Calibration is what separates a forecaster whose numbers you can trust from one who just sounds confident. It is not about being right on any single call. It is about your probabilities meaning what they say, so that a 30% from you is genuinely rarer than a 60%. Get calibrated and your percentages become a currency other people can bank on.

What calibration actually means

Suppose you make a hundred forecasts and, on each, you state a probability. Take every forecast where you said "70%." If you are calibrated, about 70% of that group came true. Do the same for your 30% forecasts: roughly 30% of them should have happened. Calibration is that promise kept, bucket by bucket, not on one prediction but across many.

This is why calibration can only be judged over a track record, never from a single call. If you say an event is 80% likely and it does not happen, you were not necessarily wrong; 80% leaves room for the other one time in five. Only when you line up all your 80% forecasts and count how many came true can anyone tell whether your 80% actually means 80%.

The calibration curve

The standard way to see calibration is a chart. Put your stated confidence on the horizontal axis, from 0% to 100%. Put the actual frequency of those events on the vertical axis. Group your forecasts into buckets (all the 10% forecasts, all the 20%, and so on) and plot one point per bucket.

A perfectly calibrated forecaster falls on the 45-degree diagonal: every bucket lands where confidence equals outcome frequency. A point that sits below the line means you were overconfident there (you claimed more certainty than reality delivered). A point above the line means you were underconfident. The shape of the curve, not any single dot, tells the story of your judgment.

The curve is the picture; a single summary number is often handier for tracking progress. The most common one folds calibration and accuracy into one score, explained in the Brier score, explained.

Calibration is not the same as accuracy

This is the distinction people miss most. Accuracy asks: how often were you on the right side of 50%? Calibration asks: did your confidence levels match reality? The two can come apart in both directions.

• A cautious forecaster who says "55% yes" on everything can be perfectly calibrated (55% of those do happen) while being almost useless, because the forecasts barely move off the coin flip.
• A bold forecaster who is right 80% of the time but always says "99%" is accurate yet badly calibrated, because the confidence is inflated.
• The ideal is both: confident when the evidence warrants it, humble when it does not, with the percentages tracking the truth either way.

Good scoring rules capture both halves. That is why the Brier score rewards you for being not just right, but right with the correct amount of confidence. A great forecaster is calibrated and decisive, pushing probabilities toward 0 and 100 only as fast as the evidence allows.

Overconfidence: the usual failure

When calibration goes wrong, it almost always goes the same way: people are overconfident. Across large studies, forecasters routinely assign 90% to things that happen more like 75% of the time, and treat "sure things" that fail more often than "sure" should allow. The gap between stated and actual confidence is the signature of overconfidence.

Overconfidence is so consistent that it shows up in nearly everyone before any training, from executives to analysts, a pattern documented at length in Douglas W. Hubbard’s How to Measure Anything, which treats calibration as a skill any estimator can and should build.

Overconfidence rarely travels alone. It rides with anchoring, confirmation, and the tendency to remember your hits and forget your misses, covered in cognitive biases in forecasting.

How to measure your own calibration

You cannot fix what you do not track. Measuring your calibration takes nothing more than a habit: write down numbers and, later, check them.

1. Log every forecast with an explicit probability. "I think it rains tomorrow" is not measurable; "70% chance it rains tomorrow" is.
2. Wait for reality to resolve each one as a clear yes or no, and record the outcome next to your original number.
3. Once you have enough forecasts (aim for at least 30 to 50), sort them into confidence buckets: your 60% calls, your 70% calls, and so on.
4. For each bucket, compute the share that actually came true and compare it to the bucket label. Your 70% bucket should land near 70%.
5. Plot the buckets against outcome frequency, or just compute a Brier score, and repeat every few dozen forecasts to watch the trend.

The first time most people do this, the pattern is stark: the high-confidence buckets underperform. That sting is the point. Seeing your 90% calls come true only 78% of the time is what actually recalibrates you.

How to train calibration

The encouraging finding is that calibration responds to practice. It is one of the few forecasting skills with a clear, fast feedback loop, and a modest amount of deliberate training moves the needle.

• Practice with feedback: make many probability estimates on questions that resolve quickly, then review where your confidence outran reality.
• Use confidence intervals for numbers: when estimating a quantity, give a 90% range wide enough that you are only surprised one time in ten. Most people start far too narrow.
• Widen your extremes: if your 90% calls keep failing, treat that as a signal to say 80% until the two line up.
• Keep score with a proper rule so you cannot fool yourself; a number you compute is harder to rationalize than a memory.

Structured programs show how far this can go. In the research behind the Good Judgment Project, a short calibration training plus regular practice measurably improved ordinary forecasters, and the best became so-called superforecasters, proof that calibration is trained, not innate.

What separates that top tier is less raw intelligence than habits like calibration and frequent updating, unpacked in superforecasters, explained.

Practice on real questions

Calibration only improves when your forecasts meet reality, again and again, with the score kept honestly. That is hard to do on your own; you need a steady stream of questions that resolve and something that remembers your numbers for you.

Clutch turns that loop into a game: you predict real news and sports with in-app Credits, every call resolves, and the app keeps score over time so you can watch your calibration improve. Get the app and start building a track record.