Your AI made your metrics look worse
Here is a conversation that happens about four months into a successful deployment.
The agent is working. It handles a decent share of incoming requests end to end. And the operations report says average handling time has gone up since launch. Somebody senior asks why the AI made the team slower, and nobody in the room has a good answer, because on the face of it the number is right there.
The number is right. The reading is wrong.
What actually happened
Agents don't take a random sample of the work. They take the easy, repetitive, well-structured end of it — which is exactly the end you wanted them to take. What is left for the humans is the long tail: ambiguous, multi-part, emotionally loaded, the ones that need a judgement call.
So the average time on human-handled work rises. It would have risen even if every person on the team got faster at their job, because the composition of what they are handling changed underneath the metric. This is selection effect, not degradation, and if you pool AI-handled and human-handled work into one average you have built an instrument that cannot tell the two apart.
Instrument them as separate streams
The fix is structural and you have to do it before launch, because retrofitting the distinction onto four months of pooled history is somewhere between painful and impossible.
Tag every unit of work with how it was handled — fully automated, assisted, or fully manual — and report the three streams separately. Then the picture decomposes into questions you can actually answer:
- Volume share. What proportion is the agent handling without a person? This is the deflection story, and it is the one most likely to be the real win.
- Time within each stream. Is assisted work faster than it used to be? Is fully manual work holding steady once you account for it being harder?
- Quality within each stream. Reopen rate, escalation rate, customer satisfaction — per stream, because an agent that is fast and wrong shows up here and nowhere else.
- Total cost to serve. The number the business actually cares about, which pools correctly even when averages don't.
Mix shift shows up everywhere
Once you start looking for it, this failure mode is not confined to handling time:
| Metric | What the agent did | How it reads if you pool |
|---|---|---|
| Average handling time | Took the short tickets | Humans got slower |
| First-contact resolution | Resolved the easy ones at contact | Human FCR collapsed |
| Escalation rate | Let simple cases bypass tier 1 | Tier 1 escalates more often |
| Satisfaction score | Handled the routine, left the fraught | Human-handled scores dropped |
Each of those is a true number and a false conclusion. Each has ended a programme that was working.
The honest version of the before-and-after
If you want a comparison that survives scrutiny, you need one of two things.
Either you hold the population fixed — compare like-for-like segments, the same category and priority before and after, which is why a baseline segmented by category and priority is worth the effort it costs to build. Or you hold the time fixed and split the population: run the agent for one group and not another, and compare across groups in the same period.
What does not work is comparing a pooled average to a pooled average across a change in mix. It feels like evidence and it is noise with a direction.
Why this matters more than it sounds
The teams that lose this argument do not lose it because the technology underperformed. They lose it because they could not explain their own dashboard in the meeting where it mattered, and a confident wrong reading beats a hesitant right one every time.
Deciding how you will separate the streams is cheap on day one. On day one hundred and twenty, it is a forensic exercise conducted while somebody is drafting the decision to switch it off.