How to Run Statistically Valid A/B Tests
You can run valid A/B tests by changing one thing, choosing one measure in advance, estimating how many visitors you need before you start, running the test for the planned time without stopping early when a version looks ahead, and recording the result. Small samples and early stopping produce false winners.
Free plan, no card needed to open it. Affiliate link: we may earn a commission at no cost to you. How we make money.
Publisher

Fix it step by step
7 clear steps, weeks per test.. Jump to the steps.
Free to read
No sign-up to read this guide or use the free resource at the end.
A no-cost place to start
Our pick is Systeme.io: a free plan with email, funnels and automation. No card needed to open it. See why.
The problem
Early results are noisy. Declaring a winner after a few conversions often means chasing chance.
Planning the test in advance keeps you honest.
This guide is for you if
- You have enough traffic to test
- You can change one element
- You can wait for the planned duration
It is not for you if
- You get very little traffic
- You stop tests early
- You test many things at once
The steps
Weeks per test.
- 1
Pick one change and one measure
Decide before you start what you change and what you measure. Plans prevent cherry-picking.
- 2
Estimate the visitors you need
Use a sample-size calculator to see how many visitors you need. Too few visitors give noisy results.
- 3
Split traffic randomly
Show each version to a random half. Fair splits keep results honest.
- 4
Run for the planned time
Run for the planned time, covering full weeks. Weekday patterns can mislead short tests. A plan you can keep beats a big one you cannot.
- 5
Do not stop early
Looking is fine; deciding early is not. Early leads often disappear.
- 6
Check the result with a significance test
Use a significance test, which many tools provide. It shows whether the difference could be chance.
- 7
Record what you learned
Write down what you learned. A record prevents repeating tests. What you measure shows what is working and what to fix, so you improve on evidence instead of guesses.
Finished the steps? Put them to work.
Free plan, no card needed to open it.
Common mistakes
- Stopping early.
- Too little traffic.
- Many changes.
- No record.
Our recommendation
Systeme.io: our pick for solo creators and small businesses starting out
Why
- Free plan with funnels, email, a course and a blog; no credit card needed.
- Paid plans start at $17/month when you outgrow 2,000 contacts.
- Funnels, email automation and checkout live in one account.
Not right for you if
- You need two-way SMS, call tracking or a shared team inbox.
- You manage several client or location accounts.
- You need advanced CRM pipelines and reporting.
Price: Free plan (2,000 contacts); Startup $17/month, Webinar $47/month, Unlimited $97/month. Source, checked 27 Oct 2026.
Alternative: If you need SMS, calls, booking and reviews for a local service business or agency, HighLevel fits better (14-day trial).
Disclosure: Unircretraites is reader-supported. If you sign up through our links, we may earn a commission at no extra cost to you. It never changes our recommendations, and we say when a tool is not right for you. How we make money
A free resource for this
Follow-up tracking spreadsheet. Free to use, no sign-up.