What Is an A/B Test? — Free Tool
An A/B test runs two versions that differ in exactly one element and compares the results. Learn why one variable is the whole discipline, how many results you need before a winner means anything, and the mistakes that produce confident but wrong conclusions.
Running two versions of a post or ad that differ in exactly one element — headline, image, call to action — and comparing results. Changing more than one variable makes the winner impossible to explain.
What it is
An A/B test, also called a split test, runs two versions of the same post or ad against each other and compares what happens. Version A is usually what you would have published anyway; version B changes exactly one thing. The point is not to find a better post, it is to learn which element moved the number, so the lesson survives past this one post.
How it is measured
You pick the metric before you run it, not after: click-through rate for a link post, conversion rate for an ad, view-through for video. Then you need enough results for the difference to mean something. A significance calculation is what separates a real difference from noise, and the smaller the gap between the versions, the more results it takes to see it at all.
Commonly misunderstood
The most common error is not changing two variables, it is stopping early. Early in a test the two versions swing past each other constantly, so if you watch and stop the moment one is ahead, you will find a winner almost every time, including when the versions are identical. The second error is testing something too small to matter: an emoji swap will not beat the noise floor at any sample size you can reach, so the test costs a week and returns nothing. Test the offer, the hook or the format before you test the punctuation.
When it matters
It matters most where you are spending money, because that is where a two-point difference compounds. On organic social the honest use is narrower: you cannot hold the audience and the timing constant, so treat organic comparisons as a source of hypotheses that a paid test can actually settle.
Related terms
Features
- Plain definition, with the one-variable rule spelled out
- Why a difference in the numbers is not the same as a difference in the versions
- How long to run a test, and why stopping at the first lead is the classic error
- What organic social can and cannot test honestly, compared with paid
- Links to the significance calculator so you can check a result rather than eyeball it
Frequently asked questions
How many variables can I change at once?
One. That is not a stylistic preference, it is what makes the result explainable: if the winning version has a new headline and a new image, the test tells you the pair won and nothing about which half did the work.
How long should a test run?
Long enough to collect a sample that would not swing on a handful of results, and at least one full cycle of your normal weekly pattern. A test that runs Tuesday to Thursday measures Tuesday to Thursday, not your audience.
When can I call a winner?
When the difference is larger than the noise, which is what a significance calculation tells you. Stopping the moment one version pulls ahead is the most common way to get a confident wrong answer, because early leads reverse constantly.
Can I A/B test organic posts?
Only loosely. You cannot show two versions to the same audience under the same conditions, so an organic comparison carries the time of day, the day of week and the algorithm along with it. Treat it as a hint, and reserve real tests for paid.
What if there is no difference?
That is a result, and a useful one. It means the element you changed is not what is limiting the post, so the next test should move to something else rather than to a smaller variation of the same thing.
