Why Most A/B Tests Are Useless
80% of tests we see in client reports ran for 3 days with 200 sessions and declared a winner at +2%. That's not A/B testing — it's statistical noise. An incorrect test is worse than no test: it gives false certainty and leads to decisions that hurt conversion rates.
Minimum Requirements for a Valid Test
95% statistical significance — at most 5% chance the result is random. Minimum 100 conversions per variant — if your site does 10 conversions/week, you need 10 weeks per test, not 3 days. One variable at a time — headline, image, or CTA button. Not all three simultaneously.
Use a sample size calculator before starting. Tools like Evan Miller's calculator tell you exactly how many visitors you need per variant. Most Romanian e-commerce sites with under 5,000 sessions/month can only validly test 1–2 elements per quarter.
Types of Tests and When to Use Each
Classic A/B test
50% of traffic sees variant A, 50% sees variant B. Best for major changes: new headline, new offer, completely different page structure. Requires the most traffic but gives the clearest signal.
Multivariate testing
Tests combinations of variables simultaneously (headline × image × CTA = 8 combinations). Needs 4–8x more traffic. Only makes sense for sites with 50,000+ sessions/month.
Split URL testing
Traffic split between two different URLs, not variants of the same page. Right for completely redesigned pages or new landing pages where you want a clean comparison.
What's Worth Testing
Focus on high-impact elements: hero headline, main offer, primary CTA, contact form length, social proof positioning, price/pricing structure. Don't waste traffic volume testing button colour, font, or spacing — the effect size is too small to measure reliably.
Classic Mistakes
- Stopping early — if after 2 days variant B shows 15% more conversions, the temptation is to declare it a winner. With low volume, a 15% difference is statistical noise. Set the test duration before starting and don't stop early.
- Running multiple tests simultaneously — each active test consumes the same traffic pool and can interfere with others.
- Ignoring seasonality — a test running Mon–Thu only is not valid. Run tests for at least 1–2 complete weeks to cover weekend variation.
Why A/B Testing Is Harder to Implement Than to Understand
The logic is simple. Execution isn't. To run a valid test you need sufficient traffic, a correctly configured tool, a clear hypothesis, and the discipline not to stop the test before reaching statistical significance. On a site with 3,000 sessions per month you can validate one element per quarter. On a site with 500 sessions per month, you practically can't test anything with certainty.
And if the test turns out negative, the interpretation is just as hard. Did variant B lose because the hypothesis was wrong, because the test ran too short, because seasonality distorted the data, or because traffic segments behaved differently? The answer to that question determines whether you test again or draw the wrong conclusion and revert to variant A without understanding why.
At DAFE Digital we test systematically for you. Well-grounded hypotheses, sufficient volume, clear conclusions without statistical noise.
Incorrect A/B testing produces false conclusions and bad decisions. We know how to build tests so you get real answers about what works — not confirmation of prior beliefs.
Adela Mincea
Marketing Economist · Fondatoare DAFE Digital · Formator ANC
Adela is a Marketing Economist with over 10 years of paid media experience across Europe, the US and Asia. She founded DAFE Digital for one reason: serious Romanian businesses deserve the same paid media expertise companies get in any other market. That's what DAFE Digital does.
Tags
