Quick answer
A shopper asking for a manager and a worker taking over are different events. In a software-support study of 5,172 human agents, AI assistance reduced requests to speak to a manager, but actual escalations were unobserved. In a separate Taobao field experiment, 7,195 of 11,069 supervised AI-handled eligible chats in the escalation analysis transferred, or 65.0%. About two thirds of those transfers were triggered by a technical limit. To evaluate checkout support, track the reason, whether a human replied, and what happened after the handoff.
Generative AI at Work · Wang et al., Agentic AI and Human-in-the-Loop Interventions
Key findings and sources
Manager requests are not recorded handoffs
Brynjolfsson, Li and Raymond’s 2025 software-support field study found a baseline manager-request rate of about 6% and an event-study decline of nearly 25% relative to that baseline after AI assistance. The authors did not observe actual escalations.
Actual escalation among selected AI-handled chats
In Wang et al.’s August 2024 Taobao field experiment, 7,195 of 11,069 supervised AI-handled eligible chats in the escalation analysis transferred, or 65.0%. The broader main sample was 680,676 chats and supplied a different denominator.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Technical triggers made up most escalations
In Wang et al.’s Taobao escalation analysis, 4,879 of 7,195 escalated chats were automatically flagged as beyond the system’s technical capabilities. That is 67.81% of escalations, or 44.1% of all 11,069 chats in the AI-handled subset as reported by the authors.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Frustration-triggered cases had more repeat contact
In a matched subgroup of Wang et al.’s Taobao study, automatic escalations triggered by frustration or doubt had a six-percentage-point higher seven-day repeat-contact rate and a 0.928-point lower customer rating than comparable human-handled control chats. Matching does not make the routing trigger randomly assigned.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Human participation differed after a transfer
In Wang et al.’s Taobao study, workers accounted for 65.4% of chat rounds in 4,879 technical escalation cases and 43.3% in 954 frustration escalation cases. These are descriptive group means in the August 2024 deployment.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Can direct and spillover effects point in opposite directions?
In the Taobao 2026 experiment, Section 5.1 reported a 0.412-point rating decline in AI-eligible chats and a 0.091-point increase in AI-ineligible chats. Rated observations were 3,865 and 14,319 in the respective regression samples.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Did human participation differ between supervisor and frustration escalations?
In Taobao’s escalation analysis, humans authored 60.1% of rounds in supervisor-initiated escalations and 43.3% in automatic frustration escalations, a 16.8-point gap. The measure is the share of chat rounds, not a share of staffing hours.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
The two handoff categories had different repeat-contact estimates
In Ni et al.’s Taobao study, the subgroup estimates versus matched human comparisons were +6.0 percentage points for frustration-triggered escalations and −3.2 points for supervisor-initiated escalations. That 9.2-point contrast is descriptive across different post-assignment groups.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Matching retained different fractions of three escalation types
In the Taobao escalation appendix, the matched sample retained 4,879 of 5,179 technical escalations (94.21%), 954 of 972 emotional escalations (98.15%) and 1,362 of 1,390 human-initiated escalations (97.99%).
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Taobao treated AI-handled chats before and after one-to-one matching in the authors’ escalation analysis. Divide the matched count by the original count separately for each reason.
Technical escalation share was slightly lower after matching
Technical escalations accounted for 68.68% of the three escalation categories before matching and 67.81% after matching in the Taobao appendix. Both shares exclude chats without escalation.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
Taobao escalation sample, one chat per observed category, before and after matching. Divide technical counts by the sum of the three escalation categories in each sample.
Compare the results
Track the customer’s request and the human takeover as separate events
The software-support field study measured when customers asked for a manager. The Taobao study observed actual escalation paths within eligible chats handled by an AI system. At Taobao, 4,879 technical triggers, 954 frustration triggers and 1,362 supervisor-initiated interventions add to 7,195 escalations. That is 65.0% of the 11,069 chats in that AI-handled analysis. The same paper’s 39,432 eligible chats in the broader 680,676-chat main sample answer a different question.
In a shop, a customer can request a person and never reach one. A supervisor can take over without a request. Log an explicit request, a routing trigger, the first reply from the receiving human, and the outcome. Report the fraction of requested handoffs that reach a person, plus the time between those events. Keep a separate measure for a system or supervisor that intervenes before the shopper asks.
The reason for the handoff changes what needs fixing
Technical limitations accounted for 67.81% of the 7,195 escalations in the Taobao analysis. A technical case can be a useful handoff: the system correctly identifies a task it cannot perform. A frustration-triggered case may arrive after several unhelpful replies, so even a correct transfer can begin in a worse position. In the study’s matched comparison, the frustration group had six percentage points more repeat contact about the same issue within seven days and a 0.928-point lower rating on a five-point scale than similar fully human-handled chats.
Do not infer that routing itself caused that difference. The reasons were selected as chats unfolded. Use them to decide where to look. For technical cases, check whether the handoff contained the requested action, affected order and verified constraints. For frustration cases, examine the earlier turns and when the assistant first had evidence it was repeating itself or had lost the shopper’s intent.
A checkout example is a shopper asking whether changing a size will invalidate a promotion. A technical handoff should say which price and inventory details the assistant verified and which rule requires human judgment. If the shopper has already received repeated guesses, the human should start by acknowledging the concrete question and give one next step. Both conversations may land in the same inbox, but they need different preparation.
Assignment to a human queue tells you little about the effort after transfer
In Taobao’s descriptive post-escalation data, humans handled an average 65.4% of chat rounds in technical escalation cases, 60.1% in supervisor-initiated cases and 43.3% in frustration cases. A 22.1-percentage-point difference between the technical and frustration categories measures participation, not an effect of adding more human replies. A short, targeted human intervention can still solve the right case.
Review examples where a shopper entered a queue and the human reply asked them to restate what the assistant already knew. A good handoff gives the recipient the current request, the facts already checked and any unresolved permissions. It also gives the customer a truthful expectation of when a person can respond. When staffing is limited, use the queue wait and the fraction receiving a real response to decide whether a new trigger would overwhelm the team.
Read containment beside resolution and repeat contact
Taobao recorded no escalation in 3,874 of its 11,069 AI-handled analysis chats. In a separate matched comparison, that no-escalation subgroup still showed a 0.858-point lower customer rating than its fully human-handled controls. No escalation means the case remained in the AI path. It does not guarantee a satisfied or resolved customer.
For a checkout pilot, reserve explicit fast exits for requests beyond the assistant’s authority, such as a buyer disputing the current price or asking to reverse an action after payment. Keep ordinary explanation questions in the standard route if the assistant can answer them with verified facts. Track resolution, repeat contact and customer acceptance separately from route counts. A higher handoff rate can be an improvement when more buyers reach the right person and avoid re-explaining a problem.
Check what the rollout did to the adjacent human queue
The Taobao experiment shows why a handoff dashboard needs a comparison outside the automated queue. In the authors’ eligibility analysis, chats in scope for AI became 16.8% shorter and ratings fell by 0.412 points. Ineligible chats became 1.8% shorter and ratings rose by 0.091 points. The rating regressions have fewer observations than the duration regressions. These estimates compare rollout effects within different eligible groups and do not tell you how an identical chat would perform under another label.
A checkout team can use the distinction by tracking two cohorts before and after a pilot. Compare the questions AI can answer and the questions it still sends directly to a person. For the latter, check time to a useful answer and accuracy before attributing improvement to fewer routine requests. Keep the reason an issue was eligible in the record. If eligibility expands to include harder cases partway through the test, report the change before comparing averages.
Count what people actually did after a handoff
In the Taobao study, humans authored 65.4% of chat rounds in automatic technical escalations, 60.1% in supervisor-initiated escalations and 43.3% in automatic frustration escalations. Supervisor-initiated cases were 16.8 percentage points above frustration cases and 5.3 below technical cases. The unit is chat rounds in each escalation type. These figures do not measure hours per worker, the average number of messages or which type was easiest to resolve.
The difference matters if you intend to staff a queue using a single transfer percentage. A conversation can remain with a machine for several turns after the shopper signals doubt, or a worker can take over immediately because an action requires permissions. Both count as transfers. They leave different amounts of context gathering, action verification and written response for the worker.
Keep the reason for the transfer, the time of the first human response and the subsequent exchanges in separate fields. Then inspect cases where a person technically owned the ticket but the shopper still had to repeat details. Compare time spent and completed resolution before deciding how many similar cases can safely stay automated.
Give supervisors a way to pick up the chat before it stalls
In the Taobao field paper, the estimated difference in seven-day same-issue repeat contact was 6.0 points for frustration-triggered escalations and −3.2 points for human-initiated escalations, each against its own matched human comparison. The point estimates are 9.2 points apart. The estimated rating differences were −0.928 and −0.524 on the five-point scale, a 0.404-point contrast. The experiment did not randomly assign one kind of escalation instead of the other. Those differences can reflect when staff stepped in and how difficult the cases already were.
The useful design decision is to make an early pickup possible. Consider a shopper whose paid order is stuck and whose first automated answer simply restates the shipping policy. A supervisor should be able to see the order status, the time sensitivity and the last thing the system tried before the customer asks for help again. In a store trial, give agents a narrow intervention reason such as a missed delivery commitment or a customer correction that did not change the next answer.
Measure time to the first human reply, completed resolution, and contact about the same problem after the pickup. Keep difficult issues visible in both groups and compare cases at a similar point in the conversation. If staff take only easy chats in one condition, a better repeat-contact number cannot tell you whether the earlier intervention worked.
Check whether the handoff sample changed before reading the result
The Taobao escalation analysis matched AI-handled chats to a control set. It retained 4,879 of 5,179 technical handoffs, 954 of 972 emotional handoffs and 1,362 of 1,390 supervisor handoffs. The category retention rates were 94.21%, 98.15% and 97.99%, respectively. Technical cases were still the largest category, accounting for 68.68% of the 7,541 escalations before matching and 67.81% of the 7,195 matched escalations.
Those small changes in the mix can matter when results differ by reason. Matching aims to make comparisons more credible on observed characteristics. The matched sample answers an evaluation question; it is not necessarily the queue that supervisors need to staff. Keep the complete queue log and the evaluation sample available separately, with the inclusion rule and period on each.
The difference in retention rates gives one useful check on generalization. If a store’s unpaired cases disproportionately include refunds or identity checks, the matched comparison may speak most clearly to its routine work. Document that difference and keep a separate tally of how the rarer cases ended before staffing an escalation queue from the analysis.
Assign a reason at the point when a person takes over
Record why the handoff happened before the final outcome is known. A specialist who takes over because the customer asks for an exception is doing different work from a specialist picking up after the model loses access to an order record. Keep both reasons if they occurred together, then give one clear routing reason to the queue so the workload report does not count the same handoff twice.
For staffing, include cases without a matched comparator and report the actual number of human rounds or handling minutes. For an outcome comparison, use the declared analysis population and acknowledge which chats were removed. If reasons or missing-data rates shift during the rollout, inspect the definitions and the excluded cases before concluding that a routing change reduced frustration.
Mark the queue entry when the agent can first act on the case, not when a routing event fires. Record whether that person has the current order state and the issue the customer wants fixed. A handoff that preserves both can be valuable even if it counts as an escalation and leaves the containment figure lower.
Asking for a person is one event. Getting one is another.
A shopper tells your recovery assistant, “Can you get someone to check if this will arrive before Saturday?” You know a shopper asked for help when that message arrives. Whether a person ever sees it is a different question. Whether that person can verify a delivery date is another. If the shopper is deciding whether to place the order now, each delay changes the answer they need.
The distinction matters when you hear that AI reduces escalations. Brynjolfsson, Li and Raymond studied AI assistance for people already working as support agents. They could see the customer's request to speak with a supervisor. The researchers explicitly did not have data on whether an escalation occurred. For a store choosing how its assistant and team work together, the evidence is useful precisely because the event is so clearly defined.
What the field study measured
The final study used 3,006,395 support chats across 5,172 agents at an enterprise software company. The authors' Figure X puts the pre-assistance share of chats requesting a manager at about 6%. Their interpretation of the estimates over time was an almost 25% decline relative to that baseline. This is evidence from a staggered rollout to human support agents who received suggestions from an AI tool.
The paper also reports a separate difference-in-differences estimate in Table IV. Its coefficient for manager requests is −0.00875 as a share of chats, equivalent to a decrease of 0.875 percentage points. That regression uses 21,839 agent-month observations. The table result and the authors' Figure X summary are two different estimates. Keep each tied to its method when citing it.
Follow one checkout question all the way through
Take the Saturday delivery question. Record when the shopper asks for a person. Record when their request reaches someone who can check inventory, destination and carrier options. Then record when that person responds and what they were able to confirm. If the store has no verified Saturday option, a useful answer can explain the delivery choices that do exist. Counting a handoff as complete when a ticket is created would miss both the wait and the final answer.
Give the person the context they would otherwise have to ask for again. That could include the checkout reference, the products, the destination detail the shopper already provided and the exact deadline. Where a detail is still missing, keep it clearly open. A team member who can pick up the question from there gives the shopper a better chance of making a decision without repeating the conversation.
A smaller request rate can have several explanations
Look at the questions the assistant answered without a transfer. If fewer people ask for a person and those replies resolve more questions within the assistant's authority, that is encouraging. If fewer ask and more leave after an unresolved answer, inspect whether the handoff was clear and easy to request. The same number on a dashboard can reflect two very different experiences.
For a pilot, define what counts as a request, a completed transfer and a resolution before you look at results. Use a clear denominator for each. The share of eligible conversations in which a shopper asks for a person and the share of routed requests a person accepts tell you where to look when something changes. Keep the timestamps beside those rates so a speedy transfer and a reply after the shopper's deadline stay distinguishable.
Study data
| Measure | Value | Context |
|---|---|---|
| Support agents | 5,172 agents | Full study includes 5,172 agents. |
| Support chats | 3,006,395 chats | Table I counts 3,006,395 conversations. |
| Baseline share requesting a manager | About 6% of chats | Authors describe the pre-assistance request rate as approximately 6%. |
| Figure X manager-request decline | About 25% relative decline | Authors describe the event-study estimates as implying an almost 25% decline relative to baseline. |
| Table IV manager-request coefficient | -0.00875 share of chats | Share-valued difference-in-differences coefficient, estimated at the agent-month level with fixed effects. |
| Table IV manager-request standard error | 0.00201 share of chats | Reported standard error, clustered at agent level. |
| Table IV manager-request observations | 21,839 agent-month observations | Regression observation count, distinct from all chat and agent counts. |
| Measure | Value | Context |
|---|---|---|
| Chats in the main Taobao analysis | 680,676 chats | All chats underlying the main worker-day analysis. |
| AI-eligible chats in the main Taobao analysis | 39,432 chats | Eligibility count in the full main sample. |
| AI-handled chats in the Taobao escalation analysis | 11,069 chats | Denominator for the four observed escalation outcomes. |
| Automatic technical escalations | 4,879 chats | Triggered when the issue exceeded the system’s capabilities. |
| Automatic escalations for frustration or doubt | 954 chats | Triggered by detected frustration or doubt. |
| Human-initiated escalations | 1,362 chats | Supervisor took over before the automated trigger. |
| AI-handled chats with no escalation | 3,874 chats | No escalation recorded in the study’s classification. |
| Customer rating difference after automatic technical escalation | -0.007 points on a five-point scale | Statistically inconclusive estimate, SE 0.089. |
| Customer rating difference after automatic escalation for frustration | -0.928 points on a five-point scale | Reported matched rating coefficient, SE 0.176. |
| Seven-day repeat-contact difference after automatic escalation for frustration | 6 percentage points | Source coefficient +0.060 for same-issue contact within seven days. |
| Customer rating difference after human-initiated escalation | -0.524 points on a five-point scale | Reported matched rating coefficient, SE 0.185. |
| Seven-day repeat-contact difference after human-initiated escalation | -3.2 percentage points | Source coefficient −0.032; reported at p < 0.1 and not at the conventional 5% threshold. |
| Customer rating difference in AI-handled chats with no escalation | -0.858 points on a five-point scale | Reported matched rating coefficient, SE 0.093. |
| Human share of chat rounds, technical escalations | 65.4% | Reported mean share 0.654, expressed as 65.4%. |
| Human share of chat rounds, frustration escalations | 43.3% | Reported mean share 0.433, expressed as 43.3%. |
| Human share of chat rounds, supervisor-initiated escalations | 60.1% | Reported mean share 0.601, expressed as 60.1%. |
| Duration effect for AI-eligible chats | -16.8% | Percent decline as interpreted by the authors. |
| Duration effect for AI-ineligible chats | -1.8% | Percent decline as interpreted by the authors. |
| Customer rating effect for AI-eligible chats | -0.412 rating points | Estimated change on a five-point customer-rating scale. |
| Customer rating effect for AI-ineligible chats | 0.091 rating points | Estimated change on a five-point customer-rating scale. |
| Pre-match AI chats with automatic technical escalation | 5,179 chats | Before matching treatment chats to the control group. |
| Pre-match AI chats with automatic emotional escalation | 972 chats | Before matching treatment chats to the control group. |
| Pre-match AI chats with supervisor-initiated escalation | 1,390 chats | Before matching treatment chats to the control group. |
| Pre-match AI chats with no escalation | 3,966 chats | Before matching treatment chats to the control group. |
What changes when you compare the figures
| Comparison | Result | Calculation |
|---|---|---|
| Table IV coefficient in percentage points | -0.875 percentage points | = Share coefficient × 100 |
| Share of Taobao AI-handled analysis chats escalating | 65% | = (4,879 + 954 + 1,362) ÷ 11,069 × 100 |
| Technical cases among Taobao escalations | 67.81% | = 4,879 ÷ 7,195 × 100 |
| Frustration cases among Taobao escalations | 13.26% | = 954 ÷ 7,195 × 100 |
| Supervisor-initiated cases among Taobao escalations | 18.93% | = 1,362 ÷ 7,195 × 100 |
| Human participation share gap, technical versus frustration escalations | 22.1 percentage points | = 65.4 − 43.3 |
| AI-eligible share of the Taobao main sample | 5.79% | = 39,432 ÷ 680,676 × 100 |
| Difference in reported duration declines by eligibility | 15 percentage points | = 16.8 - 1.8 |
| Difference in reported rating effects by eligibility | 0.503 rating points | = 0.091 - (-0.412) |
| Human participation gap, supervisor-initiated versus frustration-triggered escalations | 16.8 percentage points | = 60.1 - 43.3 |
| Human participation gap, technical versus supervisor-initiated escalations | 5.3 percentage points | = 65.4 - 60.1 |
| Difference between frustration and human-initiated retrial estimates | 9.2 percentage points | = 6 - (-3.2) |
| Difference between human-initiated and frustration rating estimates | 0.404 rating points | = -0.524 - (-0.928) |
| Technical escalation category retention in matched sample | 94.21% | = 4879 / 5179 * 100 |
| Emotional escalation category retention in matched sample | 98.15% | = 954 / 972 * 100 |
| Supervisor escalation category retention in matched sample | 97.99% | = 1362 / 1390 * 100 |
| Technical escalation share before matching | 68.68 % of escalations | = 5179 / (5179 + 972 + 1390) * 100 |
| Technical share change from pre-match to matched sample | -0.87 percentage points | = (4879 / (4879 + 954 + 1362) - 5179 / (5179 + 972 + 1390)) * 100 |
Methodology and sources
The published study examines a staggered rollout of AI suggestions to human technical support agents. Table I supplies the sample totals. Figure X and Section VII.B supply the approximate baseline and relative change. Table IV uses agent-month observations and agent-clustered standard errors. The regression coefficient was multiplied by 100 to express it in percentage points.
We distinguish QJE manager-request language from actual Taobao routing. For Taobao we retain the main sample, eligibility count and AI-handled escalation sample separately, then use Section 5.2.1 counts to calculate rates against 11,069 AI-handled chats or 7,195 escalated chats. Table 5 reports matched contrasts by observed escalation reason; the experimental assignment does not randomize the eventual reason or timing of the transfer. Ratings and repeat-contact models have different observed sample sizes, so the data rows retain each relevant count. We convert Table 6 human-round shares to percentages and keep them as descriptive means.
Download the source data (CSV)
Download the calculations (CSV)
Related reading
AI customer support productivity statistics
AI customer support quality and repeat contact statistics
Peak-season ecommerce support queue statistics by shopper question
Sources and definitions
Wang et al., Agentic AI and Human-in-the-Loop Interventions
The source data includes the population, period, denominator and limitations for every figure.