CX-301c · Module 3

Model Drift Detection

3 min read

Predictive models decay. The model calibrated on last year's data becomes less accurate this year as client behaviors change, market conditions shift, and your product evolves. Model drift is the gradual loss of predictive accuracy over time — and it happens silently. The model does not announce that it is becoming inaccurate. It simply starts making predictions that are quietly less reliable. Drift detection catches this degradation before it produces harmful decisions.

Do This

  • Track prediction accuracy monthly — compare last month's predictions against this month's outcomes
  • Set drift thresholds: if accuracy drops below 70% for two consecutive months, trigger a model recalibration
  • Monitor for systematic bias: is the model consistently over-predicting or under-predicting? Systematic bias indicates the model's assumptions no longer match reality

Avoid This

  • Assume the model stays accurate indefinitely — all models drift, the question is how fast
  • Wait for a visible failure (a major churn the model missed) to check model accuracy — by then the drift has been compounding for months
  • Recalibrate on a fixed schedule without checking accuracy — if the model is accurate, recalibration is unnecessary overhead; if it is inaccurate, the schedule may be too slow