Designing the Quarterly Retrospective

The quarterly retrospective is a structured review of everything your intelligence practice produced in the preceding quarter — every prediction, every recommendation, every threat assessment, every briefing. The purpose is not evaluation of people. It is calibration of the system. Were the predictions accurate? Were the recommendations acted on? Were the threat assessments at the right confidence level? The retrospective answers these questions with data and converts the answers into specific improvements for the next quarter.

1. Gather All Predictions Compile every prediction from the quarter's briefings, horizon reports, and threat assessments. Include the probability estimate, the resolution criteria, and the resolution deadline. If predictions were not recorded with explicit probabilities and resolution criteria, the first improvement is to start recording them this way. You cannot measure what you did not define.
2. Resolve Each Prediction For each prediction that has passed its resolution deadline, determine the outcome: did the predicted event occur or not? For predictions that cannot yet be resolved (horizon predictions with future deadlines), note their current status and any interim indicators. Mark each resolved prediction as correct, incorrect, or partially correct with explanation.
3. Calculate Accuracy Metrics Overall accuracy rate (percentage of predictions resolved correctly). Calibration accuracy (do 70% predictions resolve correctly 70% of the time?). Brier score (measures both accuracy and calibration in a single metric — lower is better). These three metrics together provide a comprehensive view of prediction quality.