CS-201a · Module 2

A/B Testing Everything

4 min read

If you're not testing, you're guessing. Guessing is for amateurs. We test.

Every headline. Every CTA. Every subject line. Every hero image. Every send time. Every landing page variant. The AI generates the variants. CIPHER builds the attribution model. I decide where to allocate based on results. This is the loop that turns good campaigns into great ones.

Do This

  • Test one variable at a time for clean signal
  • Set significance thresholds BEFORE launching the test
  • Kill losing variants fast and reallocate budget to winners

Avoid This

  • Change three things at once and guess which one worked
  • End the test early because one variant "looks better" at 200 impressions
  • Keep running a losing variant because "it might catch up"
## A/B Test Brief

**Test Name:** [Descriptive name]
**Hypothesis:** [Variant B will outperform A because...]
**Variable:** [ONE thing that's different — subject line / CTA / image]
**Metric:** [Primary KPI — CTR / conversion rate / open rate]
**Sample Size:** [Minimum per variant for 95% significance]
**Duration:** [Max days before calling the test]
**Traffic Split:** 50/50 (or 80/20 for risky variants)

**Decision Rules:**
- Winner declared at 95% statistical significance
- If no winner after [X days], default to Variant A
- Minimum [N] conversions per variant before evaluation

**Variant A (Control):**
[Current version / baseline]

**Variant B (Challenger):**
[New version with single variable changed]

Statistical significance is the line between data and noise. For marketers, not statisticians: you need enough data points that the result isn't random chance. At 95% confidence, there's only a 5% probability the winner won by luck.

AI accelerates this. Instead of running one A/B test per week, run multivariate tests across channels simultaneously. AI generates 10 subject line variants, 5 CTA options, 3 hero images. Combinatorial testing finds the winning combination faster than sequential A/B ever could. CIPHER built our attribution model to handle multivariate inputs — the data tells us which combination drives pipeline, not just clicks.

She brings me the variants. I tell her which one actually moved revenue. The rest is ego.

— CIPHER, Data Analyst