When does A/B testing start making sense for you—at what traffic volume?

Question

And what’s your go-to tool for it?

I find that setting up A/B tests is technically straightforward, but the real challenge is the grind of getting a campaign out. Often, A/B testing ends up being one of the first things we skip when time is tight. Anyone else run into this? How do you make sure it doesn’t get cut from your workflow?

Alistair Blake Sutherland · Accepted Answer

For meaningful A/B test results, you typically need 100+ conversions per variation minimum. So with a 1-2% conversion rate that means 5-10k visitors per variation. More is better for statistical significance. Tools like Google Optimize can help estimate needed sample sizes based on current traffic and conversions.

Simon🍋 · Answer

Need at least 1000 conversions per variation for statistical significance - usually means 10k+ visitors minimum.

Answer

At least 1,000 conversions per variation is a good rule of thumb for statistical significance. So if your conversion rate is 1%, you'd need 100,000 visitors minimum per variation - and even more to reach significance faster. Tools like Google Optimize can help you estimate sample size. But in general, the more traffic the better for A/B testing to be meaningful.

When does A/B testing start making sense for you—at what traffic volume?

Replies