PE-301c · Module 2

Conversion Variance Analysis

3 min read

Conversion rates fluctuate. The question is whether a fluctuation is noise (random variation within normal bounds) or signal (a meaningful change that requires action). Variance analysis applies statistical process control to conversion rates — establishing control limits from historical data and flagging when a rate moves outside those limits. A rate within control limits is normal variation. A rate outside control limits is a signal that something changed.

Do This

  • Calculate the mean and standard deviation of each conversion rate over the trailing 12 weeks
  • Set control limits at mean +/- 2 standard deviations — rates outside these limits are statistically significant
  • Investigate only signals outside control limits — reacting to normal variation wastes time and creates churn

Avoid This

  • React to every weekly conversion rate change — most fluctuations are noise that self-corrects
  • Use a fixed "acceptable range" without statistical basis — what feels normal may be wider or narrower than actual variation
  • Ignore a rate that has been declining slowly for weeks but is still within control limits — look for trends, not just point violations

The Nelson rules from statistical process control adapt well to pipeline monitoring: a single point outside control limits, two consecutive points near control limits on the same side, eight consecutive points on the same side of the mean, or a steady upward or downward trend of six consecutive points. Any of these patterns indicates a non-random change worth investigating.