How Much Does an A/B Test Developer Cost?

What drives the price of experiment development, the four pricing models, how to compare quotes, and the two costs that dwarf the hourly rate.

The cost of an A/B test developer is driven by the complexity of the variation and the state of your site, not by the platform or the hours quoted. The same hypothesis can be a two-hour build on a stable server-rendered template and a two-day build on a client-rendered checkout with four qualifying states. Any quote given before someone has looked at your actual DOM is a guess, and you should treat it as one.

We do not publish rates on this site for exactly that reason. What follows is how to work out what your own experiments should cost, and how to tell a cheap quote from an underscoped one.

The seven factors that actually move the number

In rough order of impact, largest first.

1. How your page renders

This dominates everything else. A server-rendered template where the DOM is stable at first paint is the cheap case. A client-rendered application where the variation must survive re-renders, route changes and returning to a previously visited view can be several times the effort, and needs the platform’s own re-application mechanism rather than a timer.

2. How many states the variation has to be correct in

Ecommerce is where this bites. A product page variation may need to hold up across single-variant and multi-variant products, sold-out combinations, discounted prices, gift cards, multiple currencies and languages, and guest versus logged-in sessions. Each state is a place the variation can break and therefore a place it must be checked.

3. Selector stability

Semantic markup with stable hooks is cheap to target. Page-builder output with generated class names is not, because the work includes finding something durable to bind to, or adding it.

4. Whether the change is above the fold

Above-the-fold changes bring flicker prevention into scope, and flicker prevention involves a trade-off against rendering metrics that has to be made deliberately per site.

5. How much measurement has to be built

If your analytics already emits a clean event for the action under test, instrumentation is minutes. If the goal has to be constructed, or the existing event fires inconsistently, the measurement can cost more than the variation.

6. Platform

Less than people expect, but not nothing. The developer surface differs, see Convert vs Optimizely vs VWO, and Adobe Target implementations in particular tend to carry more setup because the platform is an enterprise personalisation system rather than only a testing tool.

7. QA depth

A cross-device, cross-browser pass with targeting and goal validation is real work, and it is the line most often quietly omitted to make a quote look competitive. If two quotes differ substantially, this is the first place to look.

Four pricing models, and where each goes wrong

• Hourly. Honest and flexible, and it exposes you to scope you cannot predict. Works well when you can specify tightly, badly when the site holds surprises. Always ask for an estimate with an explicit range and what would push it to the top of that range.
• Per experiment, fixed. Easy to budget and easy to underscope. Ask what is excluded, usually QA depth, post-launch support, and any rebuild if a site deploy breaks the variation mid-run.
• Retained capacity. Predictable for both sides and the cheapest per experiment at volume, provided you can actually keep it fed. Idle retained capacity is the most expensive purchase on this list.
• Allocated hours drawn down as needed. Suits unpredictable pipelines, particularly agency work. Watch the expiry terms and whether unused time rolls over.

None is inherently better. The right one is determined by how predictable your volume is, which is why we scope per engagement rather than quoting a rate. The three shapes we work in are set out under engagement models.

Work out your own cost per experiment

Do this before you request quotes, so you have something to compare against. You need two numbers you already have.

• Take the monthly revenue flowing through the page you want to test, and divide by the number of experiments you expect to run on it in a year. That is roughly the revenue each experiment is arbitrating over.
• Take your current experiment volume and multiply by the proportion that produced a decision you acted on. Be honest; for most programmes it is well under half.

The second number is the one worth improving. Doubling the share of experiments that produce a usable answer is almost always cheaper than doubling the number of experiments, and it is entirely a function of build and QA quality.

The two costs that dwarf the rate

Experiments that produce no usable answer

The build was fine. The goal was bound to the wrong element, or fired on one arm only, or the variation was broken on iOS Safari, or the audience was wrong. Three weeks of traffic bought nothing, and the opportunity cost is every other experiment that could have used that traffic.

Decisions taken on invalid results

Worse, because it is invisible. You implement the winner permanently, the metric does not move, and the shortfall gets attributed to seasonality. A cheap developer who produces confident wrong answers is the most expensive option available, and you will not find out for two quarters.

Nobody has ever regretted paying for QA on the experiment whose result they then bet the roadmap on.

Questions to ask before accepting a quote

• What is included in QA, itemised? If the answer is the word "testing", it is not included.
• What happens if a site deploy breaks the variation mid-run, is the fix in scope?
• Who validates the goals, and do they do it on the control as well?
• Does the price include implementing the winner permanently, or is that separate? It is usually separate, and that is fine as long as it is stated.
• What would make this estimate double? A candid answer here is the single best signal you will get.
• Have you looked at our actual site, or is this from the brief? Anything quoted from the brief alone is provisional.

What we can tell you about our own pricing

We scope each piece of work after looking at the live site, and quote against that scope with the range and the risks stated. We do not publish a rate card, because the number would be either uselessly wide or misleading. If you want a figure to plan against, send one real experiment from your backlog and you will get a real one. The process is described under A/B test development.

If you are an agency reselling this, the shape is different again, and typically retained or allocated rather than per-experiment: see white-label CRO development.