Product senseAug 18, 202612 min

How to answer “our metric dropped 15%, what happened?”

This question is not testing whether you can guess the cause. It is testing whether you can narrow a search space out loud without panicking. Here is the structure I use, and the branch most candidates skip entirely.

By Lin S., founder and coach at Open Loop

Some version of this question shows up in nearly every product data science loop, and it is the round where I see the widest gap between candidates who know the material and candidates who can perform it. The material is not hard. Performing it under a timer, with an interviewer who deliberately gives you almost nothing, is a separate skill.

The failure mode is remarkably consistent. The candidate hears the prompt and immediately starts listing causes. Seasonality, maybe a competitor, maybe a bug, could be a pricing change. Every item on that list is plausible. The list itself is the problem, because a list is not a search. Nothing about it tells me you could actually find the cause on a Tuesday afternoon with a warehouse and four hours.

You are not being scored on the answer. You are being scored on whether your next question is the one that eliminates the most possibilities.

Start by refusing to answer

The prompt as given is unanswerable, and the first thing a strong candidate does is say so, politely, by asking for the four things that change everything. I want to hear these in the first ninety seconds:

01Which metric exactly. Orders and orders per active user fail for completely different reasons. If it is a rate, I want the numerator and denominator named out loud, because half of all rate mysteries are denominator mysteries.
02Over what window, against what baseline. Down 15% from yesterday, from last Tuesday, or from the same week last year? Week over week hides seasonality. Year over year hides a trend that started in March.
03What is the shape. A cliff, a slow slide, or a dip that already recovered? This single answer eliminates more hypotheses than anything else you can ask, and it is the question candidates forget.
04How wide is it. One country, one platform, one app version, or everywhere at once? Concentrated problems have findable causes. Broad problems are usually measurement, mix, or the world.

Say the interviewer tells you it is daily orders, down about 15% week over week, and it happened overnight rather than gradually. Now you have something. A cliff means a discrete change: a deploy, a logging change, a pipeline break, an outage, a config flag. It largely rules out cohort decay and competitive pressure, which do not arrive overnight.

The shape

A step change with a stable floor afterwards. The interesting question is not why is it lower but “what happened between those two days”, which is a much smaller search.

Notice the weekend pattern survives the drop. That matters. Whatever broke did not break the underlying behavior of the business, it applied a roughly constant haircut. Multiplicative and constant across days of the week is the fingerprint of a filter, not a change in demand.

The shape tells you where to look

Reference

Shape	Usually means	Check first
Cliff at a specific timestamp	A deploy, a logging change, a pipeline break, or a flag flip	The deploy log and the exact minute it moved
Gradual slide over weeks	Cohort decay, competitive pressure, SEO, or price	Retention curves split by signup month
Dip that already recovered	An incident, an outage, or you are comparing against a promo week	Incident channel and the marketing calendar
One platform only	A client bug arriving with a staged app rollout	The metric split by app version, not by platform
One country only	Payments, a local competitor, a holiday, or regulation	Payment authorization rates by country and issuer
Every metric moved together, by about the same amount	Measurement or traffic, almost never product	Raw event volume and session counts

I keep some version of this table in my head. It is not exhaustive and the mapping is not deterministic, but it puts your first three queries in roughly the right place.

The four branches

Once you have the shape, run the branches in this order. The order is the whole point: each one is cheaper than the next and eliminates more, so you never spend an hour on a segment analysis for a number that was never real.

The structure

Daily orders down 15% overnight

1. Is it real?

Did a pipeline job fail or run late?
Did anyone change the metric definition or the dashboard filter?
New app release, new SDK, new event schema?
Bot or dedupe rules changed?
Does the drop start exactly at midnight UTC or at a deploy time?

2. Is it us or the world?

Same week last year, not just last week
Holiday, weather event, major sports fixture
Competitor promo or a price move
Did total sessions fall, or only orders?

3. Where is it concentrated?

Platform, app version, geo, locale
New vs returning vs resurrected
Acquisition channel
Logged in vs guest

4. What actually changed?

Deploys and experiment ramps in that window
Config and feature flag history
Vendor or third-party incidents
Marketing spend by channel

Left to right, cheapest and most eliminating first. Say the order out loud in the interview. Interviewers cannot give you credit for structure they cannot hear.

Branch one is unglamorous and it is where the answer lives more often than anyone likes to admit. In my own experience somewhere close to a third of dramatic overnight drops are measurement artifacts. A candidate who checks instrumentation first is telling me they have been on call.

The step almost everyone skips

Before you go segment hunting, decompose the metric algebraically. Write it as a product or a sum of things you can query separately, then find out which term moved. This takes one query and it converts a vague question into a specific one.

Decompose before you segment

orders  =  sessions  ×  (carts / sessions)  ×  (orders / carts)

           traffic       browse         checkout
                         intent         completion

DAU     =  new  +  retained  +  resurrected  -  churned

revenue =  orders  ×  average order value

If sessions are flat and cart creation is flat but checkout completion fell off a cliff, you are no longer investigating orders. You are investigating a checkout page, on a specific day, and you can probably name three candidate causes without another query. That is the difference between a forty minute answer and a five minute one.

The query I would actually run first

-- Where in the funnel did it move, and when exactly?
SELECT
  DATE_TRUNC('hour', event_at)                        AS hr,
  COUNT(DISTINCT CASE WHEN step = 'session'  THEN session_id END) AS sessions,
  COUNT(DISTINCT CASE WHEN step = 'cart'     THEN session_id END) AS carts,
  COUNT(DISTINCT CASE WHEN step = 'checkout' THEN session_id END) AS checkouts,
  COUNT(DISTINCT CASE WHEN step = 'order'    THEN session_id END) AS orders
FROM funnel_events
WHERE event_at >= CURRENT_DATE - INTERVAL '14 days'
GROUP BY 1
ORDER BY 1;

Hourly, not daily. If you group by day you learn that Tuesday was bad. If you group by hour you learn it started at 14:00 and you can go read what shipped at 14:00. Say this out loud in the interview even if you are not writing real SQL, because the granularity choice is itself a signal.

When every segment looks fine

Here is the case that catches strong candidates, and it is worth practicing until you can spot it from the shape of the numbers. You split conversion by every segment you can think of and each one is flat. The blended number is still down. Nothing is broken and the metric fell anyway.

Mix shift

Both segments are perfectly stable. The blend fell 30% because the traffic mix moved toward the segment that never converted well. Nothing is broken, which is exactly why it takes people so long to find.

The cause here is usually upstream of the product entirely. Marketing turned on a broad prospecting campaign, or a partnership started sending cheap traffic, or a paid channel changed its bidding. The product is converting exactly as well as it did last month against a different audience.

This is the moment to say the phrase interviewers are waiting for: the blended rate is a weighted average, so it can move when no component moves. Then propose the fix, which is to look at the metric segment-weighted to last month's mix, or simply to report the segments separately and stop pretending the blend is a single number.

The segments people forget

Platform and geography are the ones everybody names. The list below is where I have actually found causes, and the last three come up more than you would expect.

App version, not platform. iOS is down is nearly useless. “iOS 8.42.0 is down and 8.41.3 is fine” is the answer, and staged rollouts mean the aggregate looks like a gentle slide while the real story is a cliff inside one version.
Logged in versus guest. Auth and session changes hit exactly one of these, which makes it a fast way to split the space in half.
Acquisition channel. The mix-shift case above is invisible unless you cut by channel.
Tenure, not `new versus returning`. Users in week one behave nothing like users in year two, and a bucketing that lumps them together hides cohort decay.
Payment method and issuer. For anything transactional, a single processor or a single card issuer silently raising decline rates is a genuinely common cause and almost nobody checks it in interviews.
Locale and language. A truncated translation string can break a button in one language only. I have seen this cost a real business real money for eleven days.
Device model and OS version. A layout bug that only reproduces on small screens looks like a small uniform drop until you cut by viewport.

Running it in five minutes

The script

01
Clarify the metric, window, shape, and breadth
0:00 to 1:30
Four questions, asked crisply. Then restate the problem in one sentence so you and the interviewer are solving the same thing.
02
State your structure before using it
1:30 to 2:30
“I want to check it is real, then whether it is us or the world, then where it is concentrated, then what changed.” Ten seconds, and it frames everything after.
03
Rule out measurement
2:30 to 5:00
Pipelines, definitions, releases, dedupe. Fast, and say why you are doing it first rather than treating it as box-ticking.
04
Decompose, then segment
5:00 to 15:00
Break the metric into terms, find the term that moved, then cut that term by two or three segments. Narrate what each result would mean before you get it.
05
Name a cause and how you would confirm it
15:00 to 25:00
Commit to the most likely explanation given what you found. Say what evidence would confirm it and what would kill it.
06
Recommend, then say what you would monitor
25:00 to 35:00
Immediate mitigation, the fix, and the alert you would add so nobody finds this one by dashboard again.

Rough timings for a forty minute round. Getting to a recommendation matters more than exhausting the tree, and candidates routinely run out of clock in branch one.

What separates a strong answer

I have scored a lot of these rounds, and the difference between a solid pass and a strong hire is almost never technical depth. It is four habits:

Narrating the elimination, not the hypothesis. “If sessions are flat then this is not an acquisition problem, so I would look at conversion next.” Every sentence should cut the space.
Saying what a result would mean before seeing it. This proves you are running a plan rather than reacting, and it lets the interviewer help you.
Quantifying as you go. “Checkout completion fell from 68% to 61%, which on 600k sessions is roughly the whole 15%.” Confirming the arithmetic accounts for the full drop is a step most people skip, and it catches you chasing a cause that only explains a third of it.
Ending with a decision. An investigation with no recommendation reads as unfinished. Even “I would roll back the 8.42 release today and investigate after” is a decision, and decisions are what the level ladder actually measures.

One more thing, because it is the most common unforced error I see. If the interviewer hands you a fact, use it immediately. When they say sessions are flat, that is not conversation, it is the answer to a question you asked, and you should visibly delete a branch of your tree in response. Candidates who collect facts without updating their plan sound like they are reciting, and reciting is the thing this round is designed to catch.