Skip to main content
Disclosure: Unircretraites is reader-supported. If you sign up through our links, we may earn a commission at no extra cost to you. It never changes our recommendations, and we say when a tool is not right for you. How we make money

How to Run Statistically Valid A/B Tests

You can run valid A/B tests by changing one thing, choosing one measure in advance, estimating how many visitors you need before you start, running the test for the planned time without stopping early when a version looks ahead, and recording the result. Small samples and early stopping produce false winners.

Free plan, no card needed to open it. Affiliate link: we may earn a commission at no cost to you. How we make money.

U

By Tune (AS_Tune)

Publisher

Illustration of a web page: turn visitors into enquiries
  • Fix it step by step

    7 clear steps, weeks per test.. Jump to the steps.

  • Free to read

    No sign-up to read this guide or use the free resource at the end.

  • A no-cost place to start

    Our pick is Systeme.io: a free plan with email, funnels and automation. No card needed to open it. See why.

Steps at a glance1Pick one change and one measurePick one change and one measure2Estimate the visitors you needEstimate the visitors you need3Split traffic randomlySplit traffic randomly4Run for the planned timeRun for the planned time5Do not stop earlyDo not stop early6Check the result with a significance testCheck the result with a significance test7Record what you learnedRecord what you learned

The problem

Early results are noisy. Declaring a winner after a few conversions often means chasing chance.

Planning the test in advance keeps you honest.

This guide is for you if

  • You have enough traffic to test
  • You can change one element
  • You can wait for the planned duration

It is not for you if

  • You get very little traffic
  • You stop tests early
  • You test many things at once

The steps

Weeks per test.

  1. 1

    Pick one change and one measure

    Decide before you start what you change and what you measure. Plans prevent cherry-picking.

  2. 2

    Estimate the visitors you need

    Use a sample-size calculator to see how many visitors you need. Too few visitors give noisy results.

  3. 3

    Split traffic randomly

    Show each version to a random half. Fair splits keep results honest.

  4. 4

    Run for the planned time

    Run for the planned time, covering full weeks. Weekday patterns can mislead short tests. A plan you can keep beats a big one you cannot.

  5. 5

    Do not stop early

    Looking is fine; deciding early is not. Early leads often disappear.

  6. 6

    Check the result with a significance test

    Use a significance test, which many tools provide. It shows whether the difference could be chance.

  7. 7

    Record what you learned

    Write down what you learned. A record prevents repeating tests. What you measure shows what is working and what to fix, so you improve on evidence instead of guesses.

Finished the steps? Put them to work.

Free plan, no card needed to open it.

Common mistakes

  • Stopping early.
  • Too little traffic.
  • Many changes.
  • No record.

Our recommendation

Systeme.io: our pick for solo creators and small businesses starting out

Why

  • Free plan with funnels, email, a course and a blog; no credit card needed.
  • Paid plans start at $17/month when you outgrow 2,000 contacts.
  • Funnels, email automation and checkout live in one account.

Not right for you if

  • You need two-way SMS, call tracking or a shared team inbox.
  • You manage several client or location accounts.
  • You need advanced CRM pipelines and reporting.

Price: Free plan (2,000 contacts); Startup $17/month, Webinar $47/month, Unlimited $97/month. Source, checked 27 Oct 2026.

Alternative: If you need SMS, calls, booking and reviews for a local service business or agency, HighLevel fits better (14-day trial).

Disclosure: Unircretraites is reader-supported. If you sign up through our links, we may earn a commission at no extra cost to you. It never changes our recommendations, and we say when a tool is not right for you. How we make money

A free resource for this

Follow-up tracking spreadsheet. Free to use, no sign-up.

How to Run Statistically Valid A/B Tests: questions

Use a sample-size calculator.
The planned time, covering full weeks.
Make bigger changes or use before-and-after comparisons with care.