The Complete A/B Test QA Checklist

The itemised pass we run before, at and after launch: functional, visual, responsive, cross-browser, targeting, analytics, flicker and isolation checks.

A complete A/B test QA pass checks two separate things: that the variation works everywhere real visitors will encounter it, and that the experiment is measuring what the report will claim it measures. Most teams do the first and assume the second. The second is where wasted runs come from.

This is the pass we run. Copy it, cut what does not apply to your site, and require an explicit pass or fail against each line, because "QA done" as a verbal assurance is indistinguishable from QA not done.

Before you start: get out of preview

Platform preview links normally force you into a variation and bypass the audience rules, and they can differ from how the experiment loads for a real visitor. Preview confirms the variation exists. Everything below has to be checked against the experiment running in its actual activation mode, in a clean browser profile.

• Fresh profile or incognito, with no prior bucketing cookie.
• A second clean session, to observe the other arm.
• A real device for at least the mobile pass, not only device emulation.
• Network throttling available, because several of these failures only appear on a slow connection.

1. Functional

• Every interactive element the variation adds or modifies has been clicked, tapped, keyed and submitted.
• Every element the variation did not touch, but sits next to, still works.
• Forms: validation, error recovery, double submission, and successful delivery.
• Back and forward browser navigation, and reload, all behave.
• Any element that requires authentication has been tested authenticated.
• The variation is idempotent: forcing it to apply repeatedly produces no duplicate elements and no double-bound listeners.

2. Visual

• Matches the design at each defined breakpoint, not scaled from one.
• Hover, focus, active, disabled, loading, empty and error states all present and correct.
• Long content: the longest product title, the longest translated string, the largest price.
• Zero content: empty cart, no reviews, out of stock, no search results.
• Both colour schemes if the site has a dark mode.
• No overlap with fixed furniture: cookie banner, chat widget, sticky header, iOS safe-area inset.

3. Responsive and device

• Real iOS Safari. It is the most common source of variation-specific failures and it cannot be substituted with desktop Safari.
• Real Android Chrome.
• Small viewport (around 320–375 CSS pixels wide) and large tablet.
• Landscape orientation on mobile.
• Tap targets large enough and not overlapping.
• Text remains readable at the platform’s default and enlarged font sizes.

4. Cross-browser

• Chrome, Safari, Firefox and Edge on desktop.
• The oldest browser version with meaningful share in your own analytics, not a generic support matrix.
• With an ad or content blocker active, which is a large share of real traffic and can break selectors and third-party scripts.

5. Targeting

• A qualifying session is included. Verified from the visitor side, not from the configuration screen.
• A non-qualifying session is excluded, and sees the site unchanged.
• URL rules match the live URL with query parameters, a trailing slash, and any locale or market prefix.
• The experiment does not appear on any template outside the intended set. Check the ones nobody remembers: search results, quick view, account pages, error pages.
• Exclusion or mutual-exclusivity groups behave as configured where other experiments are live on the same funnel.
• Bucketing is sticky: a returning visitor stays in the same arm.

6. Analytics and goals

The section that is skipped most and costs most. Do not infer any of it from configuration.

• Primary metric fired manually and confirmed recorded, on the control and on every variation.
• It records exactly once per action. Not zero, not twice.
• It is not also being triggered by an unrelated element matching the same selector.
• Secondary and guardrail metrics exist and have been verified before launch.
• Ecommerce events carry the expected parameters, value, currency, items, where revenue is a metric.
• The event is visible in analytics as well as in the platform, so the result can be reconciled outside the tool that produced it. GA4 DebugView is the practical tool.
• Experiment and variation identifiers are being passed to analytics if you intend to segment there.

7. Flicker and performance

• No visible flash of the control before the variation applies, checked on a throttled mobile connection rather than local broadband.
• Layout shift attributable to the experiment measured, not estimated.
• The experiment’s effect on largest contentful paint and interaction responsiveness is known and accepted.
• For asynchronous loading, whatever hiding mechanism the platform provides is configured and its timeout is sane, a hide that fails open is better than a blank page.
• The variation adds no blocking request on the critical path.

The mechanisms differ by platform and are covered in how to prevent flicker in A/B tests. Flicker is a data-validity problem before it is a performance one: a visitor who saw both experiences is not a clean sample.

8. Isolation

• Nothing outside the hypothesis has changed. Diff the control against the variation deliberately rather than trusting the intent.
• Global styles injected by the experiment do not leak into other components or other pages.
• Console is free of errors and warnings attributable to the experiment.
• No global variables or event listeners left behind after the change is applied.
• Pausing the experiment from the platform restores the original page exactly, with no residue and no deploy.

9. Regression on the critical path

• Complete a real transaction or form submission end to end, on the variation and on the control.
• Cart add, update and remove, if the site sells. On Shopify this includes after an AJAX cart update.
• Variant change on a product page, and confirm the variation is still applied afterwards.
• Login, logout and account pages if the experiment could touch session state.
• Search, filtering and pagination on any template in the experiment scope.

10. At launch

• Start at a reduced allocation if traffic allows, and confirm real sessions on both arms before going to full.
• Confirm no error-rate increase attributable to the experiment.
• Record the launch timestamp, the allocation and the intended stopping condition. Agree the stopping condition before you can see the data.

11. In flight

• Check the observed split against the configured split. A 50/50 experiment settling at 54/46 at volume is a sample ratio mismatch and means the comparison is not clean: investigate delivery, redirects, caching and bots rather than adjusting for it.
• Re-run the functional and isolation checks after any site deploy, theme release, app update or significant content edit during the run.
• Watch for a second experiment being launched onto the same funnel by someone else.
• Do not stop on a favourable reading before the stopping condition. This is the most common way a programme produces lifts that never appear in revenue.

12. After the run

• Reconcile the platform result against analytics. If they disagree materially, the result is not usable until you know why.
• Check for a sample ratio mismatch across the whole run, not just at the point you last looked.
• Check whether any other experiment overlapped in time and audience.
• State the caveats in the read-out, including anything QA flagged and any period where the variation may have been broken.
• Remove the experiment. If it won, implement the change permanently first, see stage 9 of the A/B test development workflow.

How to use this without it becoming theatre

A checklist nobody completes is worse than no checklist, because it creates a false record. Three practical rules keep it honest.

• Cut it down to your site. Delete lines that cannot apply; a shorter list that is actually completed beats a comprehensive one that is skimmed.
• Require an explicit pass or fail per line, with the person and the date. "Tested" is not a state.
• Have someone other than the author run at least sections 5 to 8. The person who wrote the code is structurally unable to see what they assumed.

That last point is most of why independent QA is worth buying at all. If you want this pass run on your experiments, including ones your own team built, that is CRO and experiment QA.