DS-301i · Module 2

A/B Testing Decision Logic

3 min read

When the logic layer has two candidate approaches — the current rules versus a new ML model, or two different ranking functions — A/B testing determines which produces better outcomes. The test: randomly assign a portion of decisions to the new logic while the majority continues with the current logic. After sufficient volume (typically 200+ decisions per variant), compare outcomes. The winner becomes the new default. A/B testing prevents two failure modes: deploying an untested logic change that performs worse, and keeping a suboptimal logic because "it works fine" without evidence that an alternative works better. The test produces evidence. The evidence drives the decision.

Do This

  • A/B test every significant logic change before full deployment
  • Run the test long enough for statistical significance — impatient testing produces random results
  • Compare on the metric that matters: outcome quality, not accept rate alone

Avoid This

  • Deploy logic changes without testing because the improvement "seems obvious"
  • Test with insufficient volume — fifty decisions per variant is noise, not signal
  • Optimize the logic for accept rate without measuring outcomes — accept rate measures comfort, not quality