How to Hire a CRO Developer: What to Screen For and Ask

A practical hiring guide: what the role requires, how to write the brief, five technical screening questions with the answers you want, and the red flags.

Hire a CRO developer by screening for measurement judgement, not just front-end skill. Ask candidates to walk you through an experiment they shipped end to end, how they isolated the variation, how they prevented flicker, how they validated the goals before launch, and what they would have caught differently in hindsight. A strong CRO developer will talk about data validity unprompted. A general front-end developer, however good, usually will not, because nobody has ever asked them to.

That single question separates the field faster than any take-home task. The rest of this article is how to build a hiring process around it.

First, decide whether you need the role at all

There is a real threshold below which a dedicated CRO developer is the wrong purchase. If you ship fewer than roughly one experiment a month, the recruitment cost, the onboarding, and the idle time will exceed the value. Below that volume the honest options are contract capacity, or improving the site on evidence without split-testing everything.

Above that volume, the constraint is almost never ideas. It is that hypotheses queue behind a product roadmap. If your experiment backlog is older than your research, you have a development problem, and no amount of additional strategy will fix it.

• Fewer than one experiment a month: use contract or white-label capacity rather than hiring.
• One to three a month: a contract specialist or a part-time arrangement usually beats a hire, because the workload swings.
• Four or more a month, sustained for two quarters: a hire starts to pay for itself, and you should still keep overflow capacity for the specialist platforms.

Write the brief around outcomes, not a technology list

The worst CRO developer job descriptions are a list of tools. Every candidate will claim all of them, and the list tells you nothing about whether they can be trusted with a result. Describe the work instead.

What a usable brief contains

• Your current experiment volume and what you want it to be.
• Which platform you are on, and whether you are willing to change it.
• What your front end actually is, server-rendered templates, a Shopify theme, a React application, because this is the single biggest driver of difficulty.
• Who owns the hypothesis, the design, and the analytics. If the answer is "the developer", say so, because that is a different and rarer person.
• Whether the role includes QA. If you do not say, candidates will assume it does not.
• What "done" means to you. If it means "the variation matches the design", you are hiring a front-end developer and should be honest about that.

A job description that lists ten tools and no responsibilities will attract candidates who list ten tools and take no responsibility.

Five screening questions, and the answers you want

These take about forty minutes together and can be run by a non-developer with a written rubric. Each has a shallow answer that sounds fine and a deep answer that tells you the person has actually shipped experiments people relied on.

1. "Walk me through the last experiment you built, from brief to read-out."

You are listening for the parts nobody volunteers unless they have been burned. Did they check the hypothesis was buildable before starting? Did they verify targeting from a real session rather than the dashboard? Did they trigger the goals manually on both arms before launch? Did they look at the split once it was live?

Weak answer: a description of the code. Strong answer: a description of the code, plus at least two things they checked that were not in the brief.

2. "How do you stop a variation from flickering, and why does it matter?"

The "why" is the real question. A candidate who describes flicker as a visual polish problem has missed the point: if part of your treatment group sees the control before the variation applies, they have been exposed to both experiences and the sample is contaminated. That is a data-validity defect.

A strong answer will also name the trade-off: the mechanisms that eliminate flicker generally cost you something in rendering metrics. It will say the decision depends on whether the change is above the fold. There is a fuller treatment in how to prevent flicker in A/B tests.

3. "Our product page re-renders when a shopper changes variant. How do you keep the variation applied?"

This is the single most reliable technical screen for ecommerce work, because it is the failure that costs real money and never shows up in preview. You want to hear that they bind to the state change rather than applying once on load, that they make the reapplication idempotent so it cannot stack or double-bind listeners, and that they verify it by actually changing a variant.

A candidate who answers "setTimeout" is telling you they have not shipped on a slow connection. Every major platform exposes a proper mechanism for this. Optimizely documents utils.waitForElement and MutationObserver-based reapplication, and Adobe Target uses view triggers. A specialist reaches for those.

4. "A test reported a 12% lift. It did not appear in revenue. What do you check?"

This question has no code in it and is the most diagnostic of the five. A good answer is a list, roughly: whether the goal was bound to the thing the hypothesis was about; whether the event fired on both arms; whether the traffic split matched configuration, which would indicate a sample ratio mismatch; whether another experiment overlapped; whether the test was underpowered and stopped when it looked good; and whether the platform and analytics were even counting the same population.

A candidate who cannot generate that list will not protect you from confident, wrong conclusions, which is more expensive than shipping nothing.

5. "What have you refused to build, and why?"

The answer you want is a hypothesis they pushed back on because it could not be measured cleanly, would not reach significance at the available traffic, or interacted with another live experiment. A specialist who has never said no to a stakeholder has either been very lucky or has been shipping tests they knew were meaningless.

Red flags

• Only ever uses the platform’s visual editor. The editor is a legitimate tool, but a developer who cannot write the variation by hand cannot handle a client-rendered page or a durable selector.
• Talks about experiments purely as design changes. Nothing about measurement, isolation or validity.
• Cannot name a single check they run before launch.
• Claims a personal win rate. Nobody controls that; it is a property of the site and the hypotheses, and quoting it is a sign of someone who does not understand variance.
• Guarantees a conversion improvement. This is not something an honest practitioner can promise.
• Has never removed an experiment after it won. Winners left running as test scripts are a permanent performance and maintenance cost.

Contract, agency or in-house?

All three are defensible. The mistake is choosing on cost per hour rather than on the shape of your demand.

• In-house is best when volume is high and steady, and when deep knowledge of your codebase compounds. It is worst when your volume swings, because a salary does not.
• A specialist contractor or development partner is best when volume is unpredictable, or when you need a platform you do not use often enough to hire for. It is worst when the work requires constant, deep context on a proprietary system.
• A full-service CRO agency is best when you also need research and strategy. It is worst when you already have those, because you will pay for them again.

If you are an agency rather than a brand, the calculation is different again, and the deciding factor is usually whether your pipeline is durable enough to keep a hire busy. That case is worked through in when a CRO agency should outsource experiment development.

Run a paid trial on one real experiment

Take-home tasks tell you whether someone can write JavaScript, which was never the question. Instead, pay for one real piece of work from your actual backlog and watch the process rather than the output.

• Did they come back with questions before writing code, or start immediately? Questions are the good sign.
• Did they flag something about the brief you had not considered?
• Did the QA evidence arrive unprompted, itemised, and in a form you could forward to a stakeholder?
• Did they tell you about a limitation, or did you find it yourself after launch?

One experiment gives you more signal than three interviews, and if it goes well you have shipped something rather than only assessed someone.

Where the money actually goes

Two costs dominate and neither is the hourly rate. The first is experiments that produce no usable answer: the build was fine, the measurement was not, and three weeks of traffic bought nothing. The second is decisions taken on invalid results, which is worse, because you then ship the wrong thing permanently and attribute the outcome to something else.

Judge candidates on how much they reduce those two, not on the rate card. What drives a quote up or down is broken down in how much an A/B test developer costs.