OC-301h · Module 2

Root Cause Analysis for AI Incidents

4 min read

Root cause analysis for AI incidents is harder than for traditional software because the failure chain often involves non-deterministic behavior. The same input does not always produce the same failure. The agent's internal state at the time of the incident may not be reproducible. The failure may have been caused by a combination of factors — a prompt change last week, a memory contamination event three days ago, and a specific input pattern that triggered the latent bug.

The investigation follows the decision record trail. Start with the bad output or wrong decision. Pull the decision record: what input triggered it? What data was loaded into context? What alternatives were considered? What confidence score was assigned? Trace each element backward: was the input correct? Was the loaded data current? Were the alternatives actually inferior? Was the confidence score justified by the evidence? The root cause is typically the first point in the chain where the answer is "no" — corrupted input, stale data, missing alternative, or unjustified confidence.

Do This

  • Follow the decision record chain backward from the bad output to the root cause
  • Check data freshness, input accuracy, and confidence calibration at every step in the chain
  • Identify contributing factors, not just the trigger — incidents usually have multiple causes

Avoid This

  • Blame the model — "the AI hallucinated" is a symptom description, not a root cause
  • Stop at the first anomaly you find — the first anomaly may be a symptom of a deeper cause
  • Skip the non-determinism analysis — if the same input does not always fail, the root cause involves state, not logic