An A/B test is only as good as how it’s run. The most common mistakes don’t show up on the dashboard: they produce a confident “winner” that turns out to be noise. This guide covers the traps that cause most false results and the simple rules that prevent them. Work out how many visitors a test needs before you start with the A/B test sample size calculator.

Quick answer: Decide your sample size and run time before you start, then don’t stop early because results look good: checking repeatedly and stopping at the first significant result can push false positives from 5% to over 25%. Run tests for whole weeks, ideally at least two, to cover weekday and weekend behaviour. Test fewer variants, check that traffic is split as planned, watch for novelty effects, and track a guardrail metric such as revenue per visitor so a win on clicks isn’t a loss on sales.

1. Peeking and stopping early

Every time you check results and could stop, you give chance another opportunity to look like a win. Optimizely's research found that uncorrected peeking can raise the false-positive rate from 5% to over 25%. Fix: set the sample size in advance and only judge the result at the end, or use a tool with sequential testing built in.

2. Tests that are too small

Small changes need large samples. With a 3% conversion rate, 95% confidence and 80% power:

Smallest lift you want to detect Visitors needed per variant
20% (3.0% → 3.6%) About 13,900
10% (3.0% → 3.3%) About 53,200
5% (3.0% → 3.15%) About 208,000

If you can't reach those numbers in a few weeks, test bigger, bolder changes or a step higher in the funnel with more traffic.

3. Too many variants

Each extra variant splits your traffic further and adds another chance of a fluke. With a control and two variants, a common correction is to test each comparison at 97.5% confidence, which raises the sample needed for a 10% lift from about 53,200 to about 64,400 per variant. Keep to one or two challengers unless traffic is very high.

4. Ignoring weekly cycles

People behave differently on Monday morning and Saturday night. Run tests in whole weeks, at least one and ideally two, and avoid launching just before a sale or holiday.

5. Sample ratio mismatch

If you planned a 50/50 split but see 52/48 on large numbers, something is wrong: a redirect, a tracking bug or bots. Results from a broken split can't be trusted. Check the split before reading any results.

6. Novelty and primacy effects

Returning visitors may click a new design just because it's new, or avoid it because it's unfamiliar. Both fade. Look at results for new and returning visitors separately and over the full run, not the first few days.

7. The wrong success metric

8. Shipping without a hypothesis

"Let's see what happens" tests teach nothing even when they win. Write down what you expect to change and why. A clear hypothesis makes losing tests useful too.

A simple pre-test checklist

  1. Hypothesis written down.
  2. Main metric and guardrails chosen.
  3. Sample size and end date set.
  4. Tracking checked with an A/A or test visit.
  5. No other big changes planned on the same pages.

When the test ends, check the result with the A/B test significance calculator and compare against conversion rate benchmarks. More tools are in the marketing tools guide.

Sample sizes calculated for a two-sided test at 95% confidence and 80% power.

Frequently asked questions

What is peeking in A/B testing?

Checking results repeatedly and stopping as soon as they look significant. It greatly increases the chance of declaring a false winner.

How long should an A/B test run?

Until it reaches the sample size you planned, and for whole weeks, ideally at least two, to cover weekly patterns.

What is sample ratio mismatch?

When the split between variants is noticeably different from what you planned, such as 52/48 instead of 50/50 on large numbers. It usually signals a technical problem.

How many variants should an A/B test have?

Usually one or two challengers against the control. More variants need much more traffic and raise the chance of a fluke.

What is the novelty effect?

Returning visitors reacting to a change simply because it is new. The effect fades, so early results can overstate a win.

Why did my winning test not improve sales?

It may have been a false positive from stopping early or a small sample, or it improved a metric that does not drive revenue.

Sources

Last updated: