CRO and Experiment QA Services
Experiment QA is the difference between a result you can act on and a number you have to argue about. It checks two separate things: that the variation works everywhere real visitors will see it, and that the experiment is measuring what the report claims it measures. We run that pass on experiments we build, and on experiments built by someone else.
CRO strategy, design and development across Shopify, WordPress, Convert, Optimizely, VWO and Adobe Target.
Does this sound familiar?
These are the situations teams actually describe when they get in touch about this, not a generic list of pain points.
A test ran for three weeks and the data is unusable
The variation was broken on one browser, or the goal fired on both arms, or the audience was wrong. None of it was visible in the platform’s preview, and all of it was findable in twenty minutes before launch.
The tracking was wrong, so the winner was not one
A goal bound to the wrong event, or firing twice on one arm. The report is internally consistent and describes something that did not happen.
Targeting sent it to the wrong people
A URL rule that matches more than intended, an audience condition that never evaluates true on mobile, or a segment that overlaps another live experiment. The comparison is contaminated before the first visitor arrives.
Flicker and layout shift are treated as cosmetic
A visitor who sees the control before the variation applies has been exposed to both. That is a data-validity defect. Layout shift on top of it costs you the visitors who scrolled past the moment it moved.
Nobody owns the pass
The strategist assumes the developer checked it. The developer assumes the client checked it. Between them, nobody opened it on an Android phone.
Experiments break silently after launch
A theme deploy, an app update or a copy change moves a selector, and the variation stops applying halfway through the run. Nothing alerts anyone; the result just gets quietly diluted.
What changes when this works
Stated as the position you end up in. No percentages: we do not publish outcome figures without the measurement conditions and the client’s agreement to it.
- Results you can present without caveats you discovered afterwards
- Broken variations caught before they cost three weeks of traffic and real revenue
- Confidence that the people who saw the treatment are the people who were supposed to
- Sample-ratio and instrumentation problems surfaced while they can still be fixed
- A defensible record of what was checked, forwardable to a client or an executive
- Fewer arguments about whether a number is real
What is included
Scope stated as specific outputs rather than as adjectives.
- Functional QA: every interaction the variation touches, exercised rather than eyeballed
- Visual QA against the intended design, including hover, focus, empty, error and loading states
- Responsive testing at real breakpoints on real devices, not a resized desktop window
- Cross-browser and cross-device matrix, with iOS Safari and Android Chrome treated as first-class
- Targeting validation: confirmation that qualifying visitors qualify and non-qualifying visitors do not
- Analytics and conversion-goal validation on control and on every variation
- Console-error check attributable to the experiment specifically
- Flicker and layout-shift check under throttled network conditions
- Variation isolation check: nothing outside the intended scope has changed
- Contamination check against other experiments running on the same audience or template
- Regression pass on the critical path: cart, checkout, forms, authentication
- Post-launch validation on live traffic, including sample-ratio checking, plus a re-check if the site is deployed mid-run
- A written pass or fail per item, so “QA done” means something specific
How the work runs
- 1
Establish what the experiment is supposed to do
We need the hypothesis, the design, the intended audience and the primary metric. QA against an unstated intention is just browsing.
- 2
Pre-launch pass
The full matrix above, run against the experiment in its real activation mode rather than in preview. Findings come back itemised, with severity and reproduction steps.
- 3
Launch check
Once live at a small allocation we confirm behaviour on genuine sessions and that both arms are recording data as expected before the experiment takes full traffic.
- 4
In-flight and post-launch validation
A re-check after any site deploy during the run, and a final pass over the data for sample-ratio and instrumentation problems before anyone reads the result as a conclusion.
What you receive
The artefacts that exist at the end, all of them yours to keep and to hand to somebody else.
- The QA report itself: every check listed, each with an explicit pass or fail
- Findings written with severity, reproduction steps and the device or browser they occurred on
- Screenshots or screen recordings for anything visual
- A targeting verification note: which sessions qualified and which did not, and how that was confirmed
- An analytics verification note per goal, per arm
- A launch-readiness recommendation: go, go with caveats, or do not launch, with the reasoning
- A post-launch validation note, including the observed traffic split against the configured one
Who this is for
Agencies who build their own tests
An independent pass by someone who did not write the code catches what the author cannot see. We can run it quietly as part of your delivery.
In-house programmes without a QA function
Experiments are shipped by whoever built them. Adding a defined pre-launch gate is usually the cheapest quality improvement available to a testing programme.
Teams with a result they do not trust
A test reported a large lift that did not appear in revenue. We audit the implementation and tell you whether the number was real.
Anyone inheriting someone else’s tests
New agency, new developer, unknown experiment layer. We document what is actually running.
Who this is not for
Saying this up front saves both of us a call. If you are on this list, the right next step is usually elsewhere on this site rather than nowhere.
- Teams who want a rubber stamp. If the experiment should not launch, the report will say so and it will say why
- Experiments with no stated hypothesis or primary metric. There is nothing to check the implementation against
- Programmes unwilling to grant analytics access. Goal validation is most of the value here and it cannot be done from the outside
- Anyone hoping QA will rescue a badly designed experiment. A clean implementation of a bad comparison is still a bad comparison
An engagement of this kind
The problem, the hypothesis and what was actually engineered. We do not publish outcome percentages until the measurement conditions can be published alongside them and the client has agreed.
Removing product-page friction and eliminating test flicker
- Problem:
- High product-page bounce on paid traffic, and a previous agency’s WYSIWYG experiments flickered for most of a second, so the results were not trustworthy either.
- Hypothesis:
- A sticky add-to-cart with above-fold trust signals, delivered as native DOM scripts, will lift add-to-cart without contaminating the sample.
- Execution:
- Rebuilt the test layer in lightweight vanilla JS on Convert Experiences and Shopify, with every GA4 event validated on both arms before launch.
Why us rather than the alternatives
Every one of these is a legitimate way to get the work done. Here is where each of them tends to break, so you can decide honestly.
Instead of the platform’s preview mode
Preview forces you into a variation, usually bypasses the audience rules, and can differ from how the experiment really loads. It confirms the variation exists. It does not confirm the right people see it or that anything is being recorded.
Instead of the developer who built it checking their own work
Not a competence problem, a structural one. The author tests the paths they had in mind while writing it, which is exactly the set of paths that will not contain the bug.
Instead of no QA at all
The alternative to twenty minutes of checking is a three-week run whose result nobody can act on, plus whatever the broken variation cost while it was live. This is the cheapest insurance in an experimentation programme.
What we test with
Named specifically, because “modern tooling” tells a buyer nothing.
- Real physical devices plus a cross-browser device cloud
- Browser developer tools: network throttling, device emulation, coverage
- GA4 DebugView and Google Tag Assistant
- Platform debug and preview modes (Convert debug, Optimizely editor and log, VWO preview, Adobe Target mbox trace)
- Lighthouse and Web Vitals for flicker and layout-shift measurement
- Playwright for repeatable regression checks where a run is long or high-risk
How this gets checked
Every engagement carries a QA pass. It is written down so that “QA done” means something specific.
- Does the variation render correctly for every qualifying visitor state?
- Does it degrade safely for visitors it was never meant to reach?
- Do the goals fire once, on the right action, on both arms?
- Is the traffic split behaving as configured?
- Does the experiment introduce any console error, layout shift or visible flash?
- Is anything outside the intended scope affected?
- Is another live experiment touching the same audience or template?
- Can the experiment be turned off cleanly, right now, without a deploy?
Ways to work together
The commercial shape is chosen after we know the scope, not before.
Dedicated partnership
Reserved capacity for a steady pipeline of work.
Flexible hours
Draw down specialist time as the work arrives.
Project engagement
A defined scope, owned end to end.
All three are described in full under engagement models.
Questions people ask before buying this
Related services
Experimentation platforms we build on
Further reading
Get My Experiments QA’d
Send us an experiment that is about to launch, or one whose result you do not believe. You will get an itemised pass or fail, not a verbal thumbs-up.