All docs

A/B testing

Configure weighted funnel variants and compare conversion results.

Updated . 4 min read.

A/B testing lets one published funnel serve different screen paths while keeping analytics tied to the same campaign URL. It is useful for testing the first screen headline, lead magnet, qualification order, or final call to action.

Set up a test in the builder

Open the funnel builder and choose the Goals & tests tab. Under A/B test:

  1. Add at least two variants. Give each a short name and a share of the visitors. Shares are relative, and the tab shows the percentage each variant really gets.
  2. Pick the goal the test is judged on: finishing the funnel, a lead, a qualified lead, a booking, or a sale.
  3. Set the smallest sample per variant, and when to decide: after a number of visitors per variant, or on a date.
  4. Choose Start test. Stop test returns every visitor to the base funnel and keeps all assignments and results, so you can start it again later.

The tab checks the same rules Formspring enforces when it saves: a running test needs two distinctly named variants with a share of the visitors, and the decision point cannot come before the smallest sample.

Tag screens with a variant name in the editor so each variant shows its own screens. While a test is stopped, screens tagged for a variant are hidden from everyone, and the tab tells you how many there are.

Judging a test

The goal, the smallest sample and the decision point are fixed before data arrives. The test is judged once, when every variant reaches the decision point, so checking early never changes the answer. Until then the Results so far panel shows each variant's visitors, successes and rate, and says that no winner can be called yet. Bots, internal traffic and test traffic are left out of every count.

Changing a share only affects visitors who have not been assigned yet. Returning visitors keep the variant they first saw.

Configuration

The tab edits the same settings the API and the MCP update tool accept. Enable testing by setting ab_test_config.enabled = true and providing a variants array. Each variant needs a stable key and a numeric weight:

json
{
  "enabled": true,
  "variants": [
    { "key": "control", "weight": 50 },
    { "key": "short-form", "weight": 50 }
  ]
}

Weights are relative. A 70 / 30 split and a 7 / 3 split behave the same way. The plan is declared next to them:

json
{
  "primary_metric": "qualified",
  "min_sample_per_variant": 100,
  "horizon_sample_per_variant": 400,
  "horizon_end_at": "2027-03-01"
}

primary_metric is one of completion, lead, qualified, booking or order. Saving a test that is switched on without two distinct variants, or with a horizon_sample_per_variant below min_sample_per_variant, is refused.

Assignment

Variant assignment uses a deterministic hash of funnel_id and visitor_id, weighted by the configured split. The same visitor on the same funnel lands on the same variant across reloads and browser tabs, which keeps the experience stable and prevents one visitor from being counted in multiple variants.

Changing variant keys after traffic starts creates a new reporting bucket. Prefer keeping the original key and changing the screen content attached to it, unless you intentionally want to start a fresh experiment.

Screen targeting

Screens without an ab_variant_key are shared across every visitor. Screens with an ab_variant_key render only for visitors assigned to that matching variant.

A common setup is:

  1. Keep the intro and outcome screens shared.
  2. Create one or more variant-specific question or offer screens.
  3. Keep the final form_collect screen shared so all variants convert through the same form fields.

Shared conversion screens make results easier to compare because each variant is measured against the same submission goal.

Reading results

Use the funnel analytics by_variant section to compare sessions, conversions, and conversion rate. Check the session count before declaring a winner, and inspect per_screen analytics to understand why a variant is winning or losing. A variant can improve early progression but hurt final submission quality if it attracts low-intent leads.

Guardrails

Avoid running multiple unrelated tests in the same funnel at once. If one variant changes the headline and another changes the form length, the analytics will not show which change caused the result. Keep experiments narrow, run them long enough to collect meaningful traffic, then archive or rename the losing screens once the winner is selected.