An A/B test compares two versions that differ in one element, to see which performs better. The idea is simple, but most ad tests fail because they test things that do not matter, or call the result before the data is in.

Start with a hypothesis, not a design

A good test starts with a sentence like:

Example: "I believe putting the returns policy in the first three seconds will raise the purchase rate, because our customers hesitate out of fear of fraud."

That format forces you to think about a real reason, and means you learn from the result even if the test fails.

What is worth testing?

Rank tests by their likely impact:

1. Angle and message (biggest impact)

Why does the customer buy? Which problem are you addressing? Which fear are you answering? At a Saudi gold retailer, changing the angle from "discount" to "safety and trust" raised average order value by 60%. See advertising high-ticket products.

2. The offer

The same product with a different offer: a bundle, a guarantee, free shipping above a threshold.

3. Format

Video or image? Someone talking or a product demo? Short or longer?

4. The first three seconds

In video, the opening decides whether the person keeps watching at all.

5. Details (smallest impact)

Button colour, or one word in the headline. These are the last things to test, not the first.

How to run the test

  1. Change one element between versions. If you change angle, format and offer together, you will not know what made the difference.
  2. Choose the metric in advance: cost per delivered order? Cost per qualified lead? Do not pick the metric after seeing the results.
  3. Give the test enough data: a small number of results can produce a random difference. Wait until each version has a reasonable number of conversions.
  4. Use the platform's tools: Meta offers an A/B test tool that splits the audience fairly between versions.
  5. Record the result: what you tested, what you found and what you will do. A test log is worth more than any single test.

When to call the result

Not after one day, and not after 10 results. Small differences on little data are usually chance. The practical rule:

  • Wait for a reasonable number of conversions per version.
  • Test for at least a full week if you can, because behaviour varies by day of week.
  • If the difference is tiny, the versions are probably equal; move on to a bigger test.

Mistakes that make a test worthless

  • Testing trivial things while the core message is untested.
  • Judging by the wrong metric: an ad with a higher CTR is not necessarily better. See the ad metrics glossary.
  • Stopping the test early at the first difference.
  • Changing budget or audience during the test.
  • Not recording results, so you repeat the same tests months later.

Testing does not stop at the ad

Sometimes the biggest improvement is entirely outside the ad: on the landing page, in the checkout steps or in signup. On a Saudi services app, removing two signup steps raised activation more than any change to the ads. See the landing page that sells.

Further reading

HubSpot's guide on how to do A/B testing explains the general method.

The bottom line

A good test is a clear question, one change, a metric set in advance and patience until the data is in. Start with the big things, angle and offer, then move to details. If you want someone to build a testing system for your ads, see the freelance media buyer page.