Most funnel A/B testing fails before the test starts, because the team tested the wrong thing. Button colours get tested because they are easy to change, not because they move numbers. Meanwhile the offer — the thing that does most of the deciding — ships untested because changing it requires a meeting.

This post gives you two things: a hierarchy for what to test first, and the statistical honesty to know whether your result means anything.

What is the funnel test hierarchy?

Test in order of leverage, not in order of convenience. The hierarchy, from highest leverage to lowest:

1. The offer. What the visitor gets at the end — demo vs free trial vs audit vs guide, and how it is framed. Nothing else in the funnel can compensate for an offer the visitor does not want. Offer tests routinely produce the kind of relative lifts (tens of percent) that design tests almost never do, and they are also the cheapest to run honestly: the difference tends to be large enough to detect with modest traffic. If you have never tested the offer, you have not started A/B testing yet.

2. The headline on the hook screen. The promise, in the visitor's words. The same offer framed as "Find your conversion leak" vs "Get a website audit" can perform very differently. Headline tests are nearly free to set up and sit directly on the highest-traffic screen, where statistical power is best.

3. The first screen's shape. What the visitor must do to start: button-only hook vs first question immediately vs short form. The first screen sees 100% of traffic, so even modest improvements compound through every screen after it — and drop-off at screen one is usually the single largest leak in the funnel anatomy.

4. Field and question count. Each field you remove typically helps completion; each field you keep buys you qualification. This is a trade to measure, not a rule to apply blindly — a B2B funnel that drops the company-size question may gain completions and lose lead quality. Judge field-count tests on qualified leads, never raw completions.

5. Design and micro-copy. Button labels, colours, imagery, screen transitions. Test these last, and only with the traffic to afford them: the true effects are small, which means they need the largest samples to detect — and small true effects plus small samples is the recipe for false positives that look like wins.

The discipline the hierarchy enforces: a team with 2,000 funnel entries a month has roughly one honest test slot per month. Spend it on the offer, not the button.

How much traffic do you actually need?

The honest math, without a statistics course:

The smaller the true difference between variants, the more conversions you need to detect it reliably. As rough orders of magnitude for a funnel converting around 10%: detecting a relative lift of ~30% needs on the order of 1,500–2,000 visitors per arm; a ~10% relative lift needs on the order of 15,000 per arm; a 5% lift, several times that again. (Evan Miller's sample-size calculator gives exact numbers for your baseline; the shape — small effects cost quadratically more traffic — is what matters.)

Three consequences worth internalising:

  • Low-traffic funnels can only honestly test big swings. Under ~1,000 entries a month, the only detectable effects are offer-sized ones. This is not a limitation of your tool; it is arithmetic.
  • Count conversions, not visitors. A funnel with 10,000 visitors and 200 conversions has 200 data points where it matters. Significance lives in the conversion column.
  • If you cannot reach the sample, do not run the test. Run the variant for everyone for two weeks and compare against your baseline judgmentally, or do five user interviews instead. A fake A/B test is worse than no test, because it launders a coin flip into a "data-driven decision".

Decide the sample size before starting, using the smallest lift you would care about. Then commit to it.

What are the false-positive traps?

The ways funnel tests produce confident wrong answers, ranked by how often we see them:

Peeking and stopping early. The classic. You check the dashboard daily and stop the moment significance appears. Checked repeatedly, a no-difference test will cross the p < 0.05 line at some point far more than 5% of the time — Evan Miller's "How Not to Run an A/B Test" works through the math showing repeated peeking can push the real false-positive rate severalfold higher. The fix: fix the sample size in advance and evaluate once, or use a sequential-testing method explicitly built for continuous monitoring.

The winner's regression. A test that "won by 28%" with a small sample will usually ship and then deliver far less. Small samples only reach significance on exaggerated observed effects, so early winners are systematically overstated. Expect shipped lifts to shrink; re-measure after shipping.

Multiple comparisons. Test four variants against control across three metrics and you have twelve chances for noise to look like signal. At p < 0.05, the expected false positives in twelve comparisons approach one — and that one gets the celebration Slack message. Fewer arms, one primary metric, decided in advance.

Mid-test changes. Editing a variant, shifting traffic split, or launching a new ad campaign mid-test invalidates the comparison — the populations before and after differ. If something material changed, restart the clock.

Simpson's paradox via traffic mix. Variant B wins overall but loses in every channel — because B happened to receive more high-intent search traffic. Funnels fed by multiple sources should randomise within source or at least check results per source before declaring anything.

Novelty and seasonality. A two-day test over a product launch weekend measures the weekend, not the variant. Run tests across at least one full week cycle; two is better.

How do you run a funnel test honestly, start to finish?

The checklist we use:

  1. Write the hypothesis down first. "Changing the offer from demo to audit will raise qualified-lead rate by ≥20%, because cold traffic resists sales calls." A test without a written hypothesis becomes whatever the dashboard says it was.
  2. One primary metric, defined in advance — qualified leads or completed funnels per visitor, not clicks on screen one.
  3. Compute the sample size, then commit. Calendar the evaluation date. Do not peek-and-stop.
  4. Randomise at the visitor level with sticky assignment — a returning visitor must see the same variant, or you are testing inconsistency. Formspring's funnel A/B testing handles weighted splits and per-variant analytics on the same campaign URL.
  5. Track conversions where they actually fire — server-side, with deduplication, so ad blockers do not silently bias one arm's measurement (the pixel vs CAPI problem applies to your own analytics too).
  6. Evaluate once, decide, document. Ship, revert, or declare inconclusive — inconclusive is a legitimate, common result. Log the test either way; a team that forgets its losing tests re-runs them annually.

What if you don't have the traffic for any of this?

A final honest note, because most funnels do not have 15,000 monthly entries: below testable traffic, qualitative methods out-earn quantitative theatre. Watch ten session recordings of people abandoning screen three. Add a one-question exit prompt ("what stopped you?"). Interview five recent leads about why they finished. Each of these reliably finds the kind of offer-sized problems that sit at the top of the hierarchy — which you can then fix with judgment and verify against your baseline trend.

A/B testing is the verification layer for funnels that already have traffic. It is not how small funnels find their first wins.

Related from this desk