The experimentation questions that trip up strong candidates
The people who fail experimentation rounds are usually not the people who forgot the math. They are the ones who could compute a p-value and then could not say what to do on Monday.
By Lin S., founder and coach at Open Loop
I have watched a lot of candidates walk into an experimentation round with a solid grasp of hypothesis testing and walk out with a no-hire. Almost never because of the statistics. The math in industry A/B testing is genuinely not that deep, and everybody preparing for these loops has read the same three blog posts about p-values.
What they get wrong is the part after the number. A test came back flat and they say so there is no effect. A test came back significant on day three and they say so we ship it. Both answers are wrong in ways that would cost a real company real money, and that is precisely what the round is for.
Nobody is hiring you to compute a p-value. They are hiring you to be the person who says do not ship that and is right.
Design questions: the six things to name
When you get “how would you test this feature”, you are being handed a checklist question. There are six items and interviewers are ticking them off. Say all six in the first three minutes and you have effectively passed the design portion.
- 01
The hypothesis, stated so it can be wrong
“Moving the payment selector above the fold reduces checkout abandonment” is testable. “Improving the checkout experience” is not. Name the mechanism you believe in.
- 02
The randomization unit
User, session, device, or something coarser like store or city. This is the decision most likely to invalidate everything downstream and the one candidates forget to state.
- 03
One primary metric
Exactly one, chosen before launch. Plus secondary metrics for understanding and guardrails for safety. If you name three primary metrics you have named none.
- 04
Minimum detectable effect and power
What size of effect matters to the business, and therefore how much traffic you need. Work from the effect that would change the decision, not from the traffic you happen to have.
- 05
Duration, with a floor
At least one full business cycle, usually two weeks, even if you hit power in four days. Weekday and weekend users are different people.
- 06
Guardrails and a stop condition
Latency, crash rate, refunds, support contacts, revenue per user. Say in advance what result makes you kill the test early.
Power, and why the number is always bigger than people expect
You do not need to derive anything. You need one rule of thumb you can produce under pressure, and you need to understand the shape of it.
The approximation worth memorizing
n per arm ~= 16 * variance / effect^2 (80% power, alpha = 0.05, two-sided)
for a conversion rate:
n per arm ~= 16 * p * (1 - p) / delta^2
worked example, 5% baseline conversion:
detect a 5% relative lift -> delta = 0.0025 -> 122,000 per arm
detect a 1% relative lift -> delta = 0.0005 -> 3,040,000 per arm
The thing to internalize is the square in the denominator. Cutting the effect you want to detect by five multiplies your sample requirement by twenty-five. This is why “we will just run it for a bit longer” is usually not a plan, and being able to say that quickly is a strong signal.
The flat result, which is the real question
If I only got one experimentation question, it would be this one, because it separates people cleanly. “You ran the test for two weeks, the primary metric moved 0.3% with a confidence interval crossing zero. What do you do?”
The weak answer is “the feature does not work, we should not ship it”. The problem is not that this conclusion is necessarily wrong, it is that it treats a failure to detect as a detection of no effect, and those are very different claims.
- Check whether you could have detected it. If the test was powered to find 2% and the interval spans minus 1.5% to plus 2.1%, you have learned that the effect is probably not huge. You have learned almost nothing about whether it is 1%, and 1% might be worth a lot of money.
- Read the interval, not the verdict. “We can rule out a loss worse than 1.5% and a gain better than 2.1%” is a real finding you can hand a product manager.
Not significantis not. - Look for offsetting effects. A flat aggregate can hide a real gain in one segment and a real loss in another. Say clearly that you are treating this as exploratory, since segment hunting after the fact inflates false positives, but a large clean split by platform or by new versus existing users is worth looking at.
- Check the funnel in between. If the feature was supposed to work by increasing add-to-cart and add-to-cart did not move at all, the mechanism failed, not just the outcome. That is a much more useful debrief for the team.
- Then make the call. Shipping a flat feature can still be right if it reduces maintenance, unblocks a roadmap, or is strategically necessary. Ship, kill, or iterate, and say which and why.
That last point is where levels get decided. A mid-level candidate reports the result. A senior candidate makes a recommendation, states the cost of being wrong, and names what would change their mind.
Peeking, and the number that makes it concrete
“It hit significance on day three, can we stop?” Almost everybody knows to say no. Far fewer can say why in a way that lands with a product manager who wants to ship.
The reason is that a fixed-horizon test controls the false positive rate at one predetermined look. Each additional look is another chance for noise to cross the threshold, and the errors accumulate.
Then give the constructive answer, because saying no is only half the job. If the team genuinely needs to look early and often, that is a legitimate requirement and there are designs for it: sequential testing, group sequential boundaries with alpha spending, or Bayesian monitoring. Naming one of those turns you from the person blocking the release into the person who can give them what they want safely.
Novelty and primacy
Any visible change to a familiar interface gets two confounds for free. Existing users click the new thing because it is new, or they struggle with it because it is not where it used to be. Both fade.
The diagnostic that gets you credit is splitting new users from existing users. New users have no prior expectation, so they show you the steady-state effect immediately. If new-user lift is 2% from week one and existing-user lift starts at 12% and decays toward 2%, you have identified novelty rather than guessed at it.
Interference, which is where the hard cases live
Standard A/B testing assumes your treatment of one unit does not affect another. Plenty of real products violate this, and if you are interviewing anywhere with a marketplace, a social graph, or shared inventory, expect to be pushed here.
| Setting | What leaks | What to do instead |
|---|---|---|
| Two-sided marketplace | Treated buyers take the supply that control buyers would have taken, so the gain is partly cannibalized | Randomize by market, region, or time slice rather than by user |
| Social or messaging product | Treated users change the experience of their untreated friends | Cluster randomization on the graph, or ego-network designs |
| Shared budget or inventory | One arm consumes a common pool and starves the other | Split the budget by arm, or switch to a time-based design |
| Learning models in the loop | A ranker trained on pooled data learns from both arms at once | Separate model instances per arm, or accept and quantify the dilution |
You are not expected to have run all of these. You are expected to notice that independence is an assumption and to say so before the interviewer has to point it out. “Before I pick a sample size, is there any way treated users could affect control users here?” is a question that visibly changes how the rest of the round goes.
The sanity check nobody asks about
Sample ratio mismatch. You planned a 50/50 split and you observe 50.6% versus 49.4% on four million users. That difference is tiny and it is also wildly implausible by chance, which means your assignment or your logging is broken.
This matters because the mechanisms that cause SRM are usually not random. A treatment that adds latency can drop slow-connection users before they get logged, and those users convert differently. Your result is now biased by an unknown amount in an unknown direction, and the only correct answer is to fix the pipeline and rerun. Bringing this up unprompted is the single most credible thing you can do in an experimentation round, because it is what people who have actually shipped tests worry about.
How to practice this
Take four tests you have actually run or read about, and for each one write half a page answering only these questions: what was the randomization unit, what would have invalidated it, what did the confidence interval let you rule out, and what did you recommend. Not what the result was. What you recommended, and what you would have needed to see to recommend the opposite.
That exercise is closer to the round than any amount of reviewing test statistics, because the round is a decision-making interview wearing a statistics costume. The candidates who figure that out tend to stop losing them.