A/B testing

Also called split testing

A/B testing shows two versions of the same thing to comparable audiences at the same time, to find out which performs better.

The method's value is that it replaces opinion with measurement. Its constraint is sample size: differences below a few percentage points need far more traffic than most creator pages receive, and calling a winner early is the most common way to conclude the opposite of the truth.

Test one change at a time or you learn that something helped without learning what. Test things that could plausibly move the number by a lot — the offer, the price, the headline — rather than button colours, where the true effect is usually smaller than the noise.

Run both versions simultaneously. Testing version A this week and B next week measures the weeks as much as the versions.

In practice

At 200 visits a week, a test between two headlines needs over a month to distinguish a 30% improvement from chance.

Common mistake

Stopping the moment one version leads. Early leads reverse constantly, and stopping at the first favourable moment is how noise gets shipped.

A/B testing

In practice

Common mistake

Read more on this