EI-301g · Module 1

Calibration Analysis

3 min read

Calibration is the alignment between your confidence levels and actual outcomes. A perfectly calibrated analyst's 60% predictions come true 60% of the time, 80% predictions come true 80% of the time, and 30% predictions come true 30% of the time. Perfect calibration is rare, but measuring your calibration reveals systematic biases that you can correct. The calibration curve plots your assigned probabilities (x-axis) against your actual accuracy at each probability level (y-axis). A perfectly calibrated analyst falls on the diagonal. Most analysts curve above the diagonal (overconfident) or below it (underconfident).

Do This

Build a calibration curve after accumulating 50+ resolved predictions — smaller samples produce unreliable curves
Separate calibration analysis by signal type — you may be well-calibrated on technology signals but overconfident on regulatory signals
Share your calibration data with your briefing consumers — transparency about your accuracy builds trust even when the numbers are imperfect

Avoid This

Calculate calibration from fewer than 30 predictions — the sample is too small and the curve will be misleading
Ignore calibration because "intelligence is not about predictions" — every briefing with a confidence level is implicitly a prediction
Adjust past predictions to improve your calibration score — this destroys the entire purpose of calibration measurement

The most common calibration finding is overconfidence — predictions assigned 80% confidence that resolve correctly only 60% of the time. The correction is simple in theory: when you feel 80% confident, assign 65%. In practice, this correction requires discipline because overconfidence feels like conviction, and lowering your stated confidence feels like weakness. It is not weakness. It is accuracy. The analyst who publishes calibrated probabilities provides more useful intelligence than the analyst who publishes confident-sounding probabilities that are systematically wrong.