How to Prevent Flicker in A/B Tests

Why flicker is a data-validity defect rather than a cosmetic one, the four causes, and the mechanism each platform gives you to prevent it.

Prevent flicker by making sure the variation is applied before the browser paints the affected content, either by loading the testing library synchronously so it executes before render, or by hiding the affected region until the variation has been applied, with a timeout that fails open. Every major platform gives you a mechanism for one or both. The decision that actually matters is which trade-off you accept, because eliminating flicker costs you something in rendering metrics.

The reason to care is not that flicker looks unprofessional. It is that flicker corrupts the sample.

Flicker is a data problem first

Flicker, also called flash of original content, happens when the original page paints before the variation is applied. Visitors in the treatment group who see that flash have been exposed to both experiences. They are no longer a clean sample of "people who saw the variation", and the difference you measure is between the control and a population that saw the control briefly followed by the variation.

There is a second cost, which is layout shift: if the variation moves content after paint, it contributes to Cumulative Layout Shift, which is a real user-experience metric with real consequences. But the sampling problem is the one that invalidates the conclusion, and it is the one that gets described as "just a bit of flicker".

If half your treatment group saw the control first, you did not run an A/B test. You ran an A versus A-then-B test, and nobody has a hypothesis about that.

The four causes

1. The library loads asynchronously

The most common cause by a wide margin. An asynchronously loaded script does not block rendering, which is the point, so the page paints while the testing library is still downloading. Whatever the library was going to change is visible in its original state until it arrives.

2. Something is deferring or moving the snippet

Even a synchronous snippet can be made asynchronous by something else. Optimisation proxies, script-deferral plugins and CDN features rewrite or postpone scripts. Convert documents the Cloudflare Rocket Loader case specifically, where the tracking code needs data-cfasync="false" so it is not deferred; the same class of problem appears with WordPress optimisation plugins and other CDNs. Loading the snippet through a tag manager has the same effect for the same reason.

3. The change is applied after paint by design

Code that waits for an element that does not exist at first paint, or waits on a promise, or runs on DOMContentLoaded, necessarily applies after something has been painted. On a client-rendered application this is unavoidable for content the framework has not rendered yet, which is why platforms provide view and re-application hooks rather than expecting you to win the race.

4. The change is expensive

A variation that loads a font, an image or a third-party widget cannot complete before paint no matter how early it starts. The fix here is to change the variation, not the loading strategy.

The two mechanisms, and the trade-off

Everything platforms offer reduces to one of two approaches.

• Execute before paint. Load the library synchronously in the head so it applies the variation before the browser renders. No flash at all, at the cost of a blocking request in the critical path, which shows up in your rendering metrics.
• Hide, then reveal. Let the page load asynchronously but hide the affected region, sometimes the whole body, until the variation is applied or a timeout expires. The visitor sees blank rather than wrong. Cheaper for the critical path, but a visible delay, and a hard dependency on the timeout failing open.

There is no third option and no free lunch. Which to choose depends on one question: does this experiment change content above the fold? If yes, the sampling problem is real and worth paying for. If the change is below the fold, the visitor will never see the flash, and paying a rendering penalty to prevent an invisible flash is a bad trade.

Scope the hiding as tightly as you can

Hiding the entire body is the blunt instrument, and it means every visitor, including the control group, pays the delay. Hiding only the container the variation modifies is better on every dimension. Whether you can depends on how early that container is identifiable.

What each platform gives you

Convert Experiences

Convert’s tracking code loads synchronously by default, which is why flicker is comparatively uncommon on it. The variation is applied before render. Asynchronous loading is supported, and Convert’s own documentation pairs it with disabling automatic body hiding and using a custom timeout with CSS to hide the body until the tracking code loads. The failure mode we see most often is a team switching to async for a Core Web Vitals score and not implementing the accompanying hiding behaviour.

Also check for interference: the Rocket Loader case above, and any optimisation plugin. More detail on the developer surface is on our Convert Experiences developer page.

Optimizely Web Experimentation

Optimizely’s documentation asks for the snippet to be placed in the head and pasted exactly as provided, without modification. That instruction is the anti-flicker strategy: the snippet is designed to execute in the right place at the right time, and wrapping it in a tag manager or deferring it defeats it. For content that does not exist at first paint, use utils.waitForElement or utils.waitUntil rather than a timer, and rely on the platform’s MutationObserver-based re-application on dynamic sites instead of racing the framework.

More on that surface on our Optimizely developer page.

VWO

VWO’s SmartCode is asynchronous by default, which is the better default for page speed and the reason above-the-fold VWO campaigns flash unless something is done about it. VWO documents a synchronous SmartCode option for exactly this case. The practical rule: if most of your campaigns are below the fold, stay asynchronous; if you are consistently testing heroes and headlines, the flash is a data-quality problem and the synchronous option is worth its cost.

See our VWO developer page for the rest of the implementation surface.

Adobe Target

Adobe documents a prehiding snippet for asynchronous at.js deployments, whose entire purpose is to prevent flicker while content is being fetched. There is an important nuance for single-page applications: when you use adobe.target.triggerView() to show targeted content in a view, at.js 2.x manages flicker for you by pre-hiding the location where the view needs to be shown, so you do not add prehiding logic manually for those views. Prehiding is about the initial page load; views are handled by the library.

Details on our Adobe Target developer page.

Client-rendered applications

On a React or similar front end you cannot generally apply the change before paint, because the content does not exist at first paint. Fighting for it produces brittle code. The workable approach is different:

• Use the platform’s own hook for content readiness and re-application rather than a timeout.
• Make the change idempotent, so re-applying on a re-render cannot duplicate elements or double-bind listeners.
• Where the platform supports a view concept, use it. That is what it is for, and it is where flicker handling is built in.
• Scope any hiding to the component, never the body, because a SPA body-hide is a blank screen.

How to measure it, rather than argue about it

Flicker is a timing bug, so it disappears exactly where developers look for it: a fast machine on a fast connection. Measure it deliberately.

• Throttle the network and the CPU in developer tools. Slow 4G and a 4× CPU throttle is a reasonable floor, and closer to your actual mobile traffic than your laptop is.
• Record a performance trace or a screen recording and step through frames. Flicker is a small number of frames; the eye is not a reliable instrument at that resolution.
• Measure layout shift attributable to the experiment, control versus variation, rather than estimating it.
• Test on a real mid-range Android device. Emulation does not reproduce the CPU cost of parsing and executing the library.
• Repeat with a cold cache. Warm-cache testing hides the problem entirely.

When to accept it

Sometimes the right answer is to leave it. If the change is below the fold, if the affected element renders late for everyone anyway, or if the only available fix would blank the page for every visitor including the control, the cure is worse. What is not acceptable is not knowing, an unmeasured flash on an above-the-fold experiment is a caveat that belongs in the read-out.

On Shopify specifically there are additional causes, because the theme re-renders parts of the product page on variant change and updates the cart over AJAX. That case is covered in the hidden cost of A/B test flicker on Shopify.