The Complete A/B Test Development Workflow

Nine stages from brief to rollout, what "done" means at each one, and the specific places experimentation programmes leak value.

The A/B test development workflow has nine stages: technical review of the brief, selector and state mapping, variation build, targeting configuration, measurement instrumentation, pre-launch QA, launch validation, in-flight monitoring, and rollout or removal. The stage teams skip is the first one, and skipping it is what causes most of the failures in the other eight.

Each stage below has a definition of done and an artefact. If a stage produces no artefact, it did not happen. That is not pedantry. It is the only way a programme stays auditable when someone asks in three months why a decision was made.

Stage 1, Technical review of the brief

Before any code, read the hypothesis and the design against the live site. The question is not "can this be built" but "can this be built such that the result means something".

What to check

• Does the target element exist on every template named in the brief, and only those?
• Can the intended audience actually be expressed in the platform, from data available client-side at decision time?
• Does the primary metric exist as a reliable event today, or does it need building?
• Is anything else running on the same funnel that could interact with this?
• Is there enough traffic on the targeted pages for the expected effect size to be detectable in a reasonable window?

Done when: you have a written scope, the risks named, and any question that would change the build answered. Artefact: a scope note on the ticket. The most valuable output of this stage is occasionally "do not run this experiment", and a workflow that cannot produce that conclusion is not a workflow.

Stage 2, Selector and state mapping

Establish what you are binding to and every state it must be correct in. On ecommerce this is where the real complexity lives, and doing it on paper is far cheaper than discovering it in QA.

• Choose selectors for durability, not convenience. Semantic attributes and stable ids over generated class names. Where you control the markup, add a dedicated hook.
• Enumerate the states: logged in and out, single and multi variant, in stock and sold out, discounted, multiple currencies and languages, empty and populated cart, first visit and returning.
• Identify what re-renders. On Shopify, variant selection re-renders parts of the product form and the cart updates over AJAX. On a client-rendered application, routes and components re-render independently.

Done when: you have a list of selectors and a list of states. Artefact: both, on the ticket. This list becomes the QA matrix later, which is why writing it now costs nothing.

Stage 3, Build the variation

Now write the code. Three properties matter more than elegance.

• Idempotent. Running the change twice must produce the same result as running it once, no duplicated elements, no double-bound listeners. On a re-rendering page it will run many times.
• Reversible. Pausing the experiment from the platform must leave no residue in the DOM and require no deploy.
• Isolated. Nothing outside the hypothesis changes. Resist the tidy-up you notice on the way past; raise it separately.

Separate shared setup from per-variation changes using whatever the platform provides for it, Convert distinguishes Global Experience JS from Variation JS, and Optimizely has project-level JavaScript alongside per-variation code. Duplicating setup into every arm is how arms drift apart by more than the thing under test.

Done when: the change is correct in every state from stage 2, on the developer’s machine. Artefact: the code, commented, in version control or the platform.

Stage 4, Targeting

Configure audience and URL rules, then verify them from outside the dashboard. The dashboard tells you what you configured, not what happens.

• Test from a qualifying session and confirm the visitor is included.
• Test from a non-qualifying session and confirm they are excluded.
• Check URL rules against the live URL including query parameters, trailing slashes, and any locale or market prefix.
• Check exclusion or mutual-exclusivity groups if other experiments are live on the same funnel.

Done when: both a qualifying and a non-qualifying session have been observed behaving correctly. Artefact: a note of what was tested from where.

Stage 5, Measurement instrumentation

The most-skipped stage with the highest cost of being wrong. Every goal in the experiment must be triggered by hand and confirmed recorded, on the control as well as on each variation.

• Fire the primary metric manually on every arm. Confirm it records once, not zero times and not twice.
• Check the event is not also firing from something unrelated that matches the same selector.
• Confirm secondary and guardrail metrics exist before launch, not after a result looks strange.
• Verify in analytics as well as the platform, GA4 DebugView is the practical tool for this, so the result can be reconciled outside the tool that produced it.

Done when: every metric has been fired and observed on every arm. Artefact: a list of metrics with a tick against each arm.

Stage 6, Pre-launch QA

The full pass, run against the experiment in its real activation mode rather than in the platform’s preview. Preview forces you into a variation and usually bypasses the audience rules; it proves the variation renders and nothing else.

The complete itemised pass is published as the A/B test QA checklist. Done when every item has an explicit pass or fail. Artefact: the completed checklist, which is also the thing you forward to a stakeholder instead of the word "tested".

Stage 7, Launch validation

Launch at a low allocation first if the platform and your traffic allow it. Then confirm on genuine sessions, before the experiment takes full traffic:

• Both arms are recording data.
• The observed split is tracking towards the configured split.
• No error rate increase attributable to the experiment.
• The variation is rendering for real visitors, not just for you.

Done when: real sessions have been observed on both arms. Artefact: a timestamped launch note.

Stage 8, In-flight monitoring

Experiments break during the run and nothing alerts anyone. The two things to watch are the split and the site.

• Sample ratio: if a 50/50 experiment settles at 54/46 at volume, something is wrong with delivery or counting. Investigate rather than adjusting for it: the usual culprits are a redirect, a caching layer, a bot, or an instrumentation fault.
• Any site deploy during the run means re-checking the variation. A theme release, an app update or a content edit can move a selector and silently end the experiment.
• Do not stop the test the day it looks good. Peeking and stopping on a favourable reading is the most common way programmes manufacture lifts that never appear in revenue.

Done when: the experiment reaches its pre-agreed stopping condition. Artefact: a monitoring log, however brief, including any re-checks after deploys.

Stage 9, Rollout or removal

A finished experiment is not a finished job.

• If it won: implement the change permanently in the codebase or theme, then remove the experiment. A winner left running as a test script is a permanent performance and maintenance cost with no remaining upside.
• If it lost: remove it. Also record why you thought it would win, because that is the reusable output.
• If it was inconclusive: record what you would need, more traffic, a larger change, a different metric, to answer the question. Inconclusive is a result, and the most commonly wasted one.

Done when: nothing about the experiment is still executing on the site. Artefact: a read-out that states the caveats plainly, including anything QA flagged.

Where programmes actually leak

Across the programmes we have picked up mid-flight, the leaks cluster in the same four places, and none of them are the build.

• No stage 1. Experiments start with code, so unbuildable briefs are discovered in QA and unmeasurable hypotheses are discovered at read-out.
• No stage 5. Goals are assumed correct because they were configured, and the arms were never both checked.
• No stage 8. Nobody re-checks after deploys, so an unknown share of the run measured nothing.
• No stage 9. Old experiment scripts accumulate, the site gets slower, and nobody is sure which are still needed.

A programme that ships ten well-built experiments a quarter and can only trust four of them does not have a development problem. It has a stages 1, 5, 8 and 9 problem.

If you want this workflow run on your experiments, that is A/B test development. If you only want stages 6 to 8 on experiments your own team builds, that is CRO and experiment QA.