A/B testing with low traffic
Before you split a page with a few thousand visitors a month, work out how large a change the test can see. Usually it is larger than the change you were planning.
By Daria Dovzhikova · October 2026
The short version
- At 2,000 visitors a month and a 5% conversion rate, a four-week test has an estimated 80% power for a lift of about 62%.
- Power depends on sample size, baseline rate and effect size. Sparse conversions can leave wide uncertainty.
- With low traffic, test bigger changes, measure a higher-volume step, read the sessions, or run a before-and-after with the decision rule written first.
- Report the counts next to every rate.
Planning lift for 80% power in four weeks
Treat the monthly traffic below as the number of eligible independent randomized observations available in a fixed four-week window, split 50/50 between control and variant. Each cell estimates the positive relative lift needed for 80% power at two-sided alpha 0.05, rounded to the nearest percentage point. It is not a detection boundary: smaller effects can be significant, and larger ones can be missed.
| Eligible observations / 4 weeks | Per arm | 2% baseline | 5% baseline | 10% baseline |
|---|---|---|---|---|
| 1,000 | 500 | 166% | 92% | 59% |
| 2,000 | 1,000 | 108% | 62% | 41% |
| 5,000 | 2,500 | 63% | 37% | 25% |
| 20,000 | 10,000 | 30% | 18% | 12% |
| 100,000 | 50,000 | 13% | 8% | 5% |
These are two-sided two-proportion z-test normal-approximation planning estimates, using a pooled proportion for the null variance and separate control and variant proportions for the alternative variance, as documented in statsmodels. We solve for the lift whose two-tail rejection probability is 0.80. The estimates assume one prespecified comparison, independent randomized observations and a fixed horizon, with no repeated peeking or multiple-comparison adjustment.
At 500 observations per arm and a 2% baseline, the control has only 10 expected conversions. Normal approximations can be inaccurate with sparse expected counts; check the planned analysis with an appropriate exact method or simulation before committing to a test. Repeated visits by the same person are not independent observations. The A/B test calculator uses a simpler baseline-variance approximation, so its estimates can differ, especially for large lifts.
Count conversions, not visitors
Read the table by conversions. At 2,000 visitors a month and a 5% rate, each arm expects about 50 conversions under no effect. At a 2% rate it expects about 20. Counts of 18 and 24 in equal-sized arms look like a 33% relative lift, but the estimate is imprecise. Report the effect estimate and its uncertainty alongside the counts.
That is why a rate on its own misleads at this size. Write "24 of 1,000", not "2.4%". Anyone reading the result can then see how little it rests on.
What to run instead
01
Test a change big enough to show up
If the table estimates a 62% lift for 80% power, test changes that could plausibly produce one: a different offer, a different price, a removed onboarding step, a new entry point. Small changes may need much more traffic than you have. Write the expected size of the effect on the card before you start.
02
Measure a step with more events
Detectable lift depends on how many conversions each arm collects. A page where 10% of visitors click through to signup gives you five times the events of one where 2% pay, so the same traffic can detect a much smaller change. Use the upstream step as the primary outcome only if it predicts the one you care about, and say so on the card.
03
Read the sessions instead of counting them
With few conversions per arm, rate estimates can have wide uncertainty. There is no universal fifty-conversion cutoff. Watch the recordings of the visitors who left at the step you want to change, read the support questions, ask the last five people who converted what almost stopped them. That evidence can identify a broken step; it cannot prove a lift, and the decision log should say which one you have.
04
Run a before-and-after with a written decision rule
When a split test is out of reach, change the page for everyone and compare a fixed period before with a fixed period after. It is weaker evidence: pricing, launches and seasonality move the numbers too. Make it honest by writing the dates, the outcome, and the result that would make you continue, change or stop before the change ships, and by recording anything else that changed in the same weeks.
Write it down before it starts
Whichever of the four you run, the discipline is the same: one primary outcome, a baseline with its source, a fixed review date, and the result that would make you continue, change or stop. Decide those before the change ships, then do not check the result daily and stop when it looks good. The experiment card is a one-page format for exactly this, with a copyable template.
An inconclusive result means the evidence did not resolve the comparison. It does not establish that the true effect is zero or below the planning lift. Inspect the effect estimate and its uncertainty, together with the cost of a wrong decision, before choosing to stop, redesign, or plan more data. The ASA statement on p-values explains why a significance threshold alone is insufficient for a business decision.
Questions
How much traffic do you need for an A/B test?
It depends on the baseline conversion rate and the smallest lift you care about. Using the two-sided two-proportion normal planning method below, with alpha 0.05 and 80% power, a 20% relative lift needs about 8,160 independent observations per arm at a 5% baseline, or 21,110 per arm at 2%. Those are fixed-horizon planning estimates, not guarantees.
Can you A/B test with 1,000 visitors a month?
With 500 independent observations per arm in four weeks, the planning estimate for 80% power is roughly a 92% relative lift at a 5% baseline, or 166% at 2%. Smaller effects have lower power, so an inconclusive result is more likely. Low expected counts also make the normal approximation less reliable. Test bigger changes, measure a higher-volume step, or use qualitative evidence and a written before-and-after.
Should I run a low-traffic test for longer?
Running longer adds visitors, but it also adds time for other things to change: pricing, product, traffic sources, season. Decide the duration before you start, from the sample size, and do not extend a test because it has not reached significance yet. If the required duration is several months, weigh the cost and changing conditions against testing a larger change or using another source of evidence.
Is a Bayesian test better for low traffic?
A Bayesian analysis can report posterior probabilities and credible intervals, conditional on its model and prior. It does not create information that small samples do not contain. With sparse data, conclusions can depend strongly on the prior and effect size. Specify the model, prior, stopping rule and decision costs before starting; neither approach guarantees a clear answer.
The Growth Pilot
Four weeks, two real growth experiments, a decision every Friday.
Low traffic is the normal case for a seed-stage developer tool, and the pilot's two cards are sized to it. For developer-tool and AI-product founders who don't have a growth lead yet. $5,000 fixed, half at kickoff. How a pilot runs · What an experiment card looks like
Book a 30-minute call