DS-301g · Module 2

Probability Calibration

3 min read

A model that says "this deal has an 80% probability of closing" must be right 80% of the time for deals it scores at 80%. If it is actually right 60% of the time, the model is overconfident. If it is right 95% of the time, the model is underconfident. Calibration is the alignment between predicted probability and actual frequency. The calibration curve plots predicted probability on the x-axis against observed frequency on the y-axis. A perfectly calibrated model falls on the diagonal. Most models are not calibrated out of the box — they need post-hoc calibration using techniques like Platt scaling or isotonic regression. Uncalibrated probabilities are misleading. The executive who trusts an 80% score that is really 60% makes overconfident decisions.

Do This

  • Check the calibration curve for every probabilistic model before deployment
  • Apply calibration correction if the model is consistently overconfident or underconfident
  • Re-check calibration quarterly — model calibration drifts as the underlying data distribution changes

Avoid This

  • Deploy a probability model without checking calibration — uncalibrated probabilities mislead decision-makers
  • Assume a high accuracy model is well-calibrated — accuracy and calibration are different properties
  • Calibrate once and never re-check — the world changes, the calibration drifts, the decisions degrade