Quick answer
In Taobao’s supervised AI chat sample, humans contributed 65.4% of chat rounds on average after technical escalations, compared with 43.3% after frustration-triggered escalations. That is a 22.1-point gap. The frustration group also had a 6-point higher seven-day repeat contact rate than its matched human-only comparison group.
Key findings and sources
About 65% of the supervised Taobao sample escalated
Offload Rep’s count of Wang et al.’s published groups finds 7,195 escalations among 11,069 supervised AI-eligible Taobao chats, or about 65.0%. Frustration-triggered cases accounted for 954, or 13.3% of those escalations.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
11,069 AI-eligible Taobao chats supervised by treated workers in August 2024; post-handoff effort descriptives use the three escalation groups. Add technical, emotional and human-initiated counts; divide separately by the supervised total and by the escalation total.
The human share of rounds differed by 22.1 points
After technical handoffs in Taobao’s August 2024 sample, the mean human share of chat rounds was 65.4%, versus 43.3% after frustration-triggered handoffs. The arithmetic gap is 22.1 points.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
11,069 AI-eligible Taobao chats supervised by treated workers in August 2024; post-handoff effort descriptives use the three escalation groups. Convert the two mean proportions to percentages and subtract. These are post-escalation descriptives.
Human replies averaged 2.302 seconds longer after frustration-triggered alerts
In Taobao’s observed post-handoff chats, the mean human response delay was 20.965 seconds in frustration-triggered cases and 18.663 seconds when a supervisor took over first, a difference of 2.302 seconds.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
11,069 AI-eligible Taobao chats supervised by treated workers in August 2024; post-handoff effort descriptives use the three escalation groups. Subtract the published subgroup means. The escalation categories were not randomized.
Matched rating declines differed across handoff groups
Taobao’s frustration-triggered handoffs had a matched rating difference of minus 0.928 points. Supervisor-initiated handoffs had a matched difference of minus 0.524 points, a 0.404-point arithmetic gap. The matched groups cannot establish the effect of changing the handoff trigger.
Wang et al., Agentic AI and Human-in-the-Loop Interventions
11,069 AI-eligible Taobao chats supervised by treated workers in August 2024; post-handoff effort descriptives use the three escalation groups. Compare the two Table 5 customer rating coefficients, each relative to its own human-only controls.
Compare the results
Where supervised AI chats needed a handoff
Separating escalation types shows which problems the human team actually received.
Send the unresolved question with a technical handoff
A shopper who cannot select a delivery option is in a different conversation from a shopper who has repeated the same concern and no longer believes the next answer. Both may need a human, but the receiving agent needs different information. For the delivery case, send the cart state, options the checkout actually showed and the action that failed. The agent can investigate the missing service without asking the shopper to reconstruct every screen.
The Taobao authors recorded 4,879 algorithm-triggered technical escalations and 954 frustration-triggered escalations among 11,069 AI-eligible chats handled by the system with human supervision. The three types of handoff together accounted for about 65% of this particular sample. These were marketplace service chats, including work outside the checkout moment. The percentages describe the study population, so a DTC team should size its queue using its own escalation rates.
Show a human what went wrong before the shopper repeated it
The human share of rounds averaged 65.4% after technical escalations and 43.3% after frustration-triggered ones. Humans also sent about one fewer message on average in the frustration group. These differences describe how the conversations unfolded. A shorter human share can reflect a longer earlier AI exchange as well as differences after takeover. Increasing message count for its own sake would miss the decision.
Give the receiving agent a short chronology, the concern the shopper already raised and the promise or charge that remains unverified. A human who sees the issue was raised twice can acknowledge it once and work on the remaining decision. Record when the shopper first asks for help, when a human becomes available and when the shopper receives an answer they can use. An internal assignment timestamp captures only one part of that sequence.
Judge the handoff by what the shopper does next
In the study’s matched comparison, the frustration-triggered group had a 6-point higher repeat-contact rate within seven days and a customer rating 0.928 points lower than its human-only comparison group. It is possible that harder conversations were more likely to enter that group. Treat those figures as a reason to inspect the route separately, rather than as a prediction for your team or proof that one alert caused the outcome.
For a store test, label the reason for handoff before reviewing the outcome. Read first useful human reply, repeat contact on the same issue and checkout completion from the original eligible group. Check whether the shopper received an answer before the purchase window closed. That last detail distinguishes a useful intervention from a transfer that happened after the customer had already moved on.
Consider a shopper who was told twice that delivery by Friday should be possible, then sees only a Monday estimate at checkout. The handoff should make both dates visible, along with the source of the Monday quote. The receiving agent can check the eligible options, explain which date the store can verify and offer a choice the shopper can still use. If no eligible option remains, the shopper deserves a clear answer before another reminder sends them back to the same cart.
Study data
| Measure | Value | Context |
|---|---|---|
| Supervised AI-eligible chats | 11,069 chats | Total across technical, emotional, human-initiated and no-escalation groups. |
| Algorithm-triggered technical escalations | 4,879 chats | Monitoring detected an issue beyond the AI system’s ability. |
| Algorithm-triggered emotional escalations | 954 chats | Monitoring detected signs of frustration or doubts. |
| Human-initiated escalations | 1,362 chats | A supervisor took over before the algorithm initiated the escalation. |
| Chats resolved without escalation | 3,874 chats | The agentic system handled the chat without a human taking over. |
| Human round share after a technical escalation | 65.4% | Published ratio 0.654 converted to a percentage. |
| Human round share after an emotional escalation | 43.3% | Published ratio 0.433 converted to a percentage. |
| Human round share after a supervisor takeover | 60.1% | Published ratio 0.601 converted to a percentage. |
| Human reply delay after a technical escalation | 20.006 seconds | Mean average human response delay following escalation. |
| Human reply delay after an emotional escalation | 20.965 seconds | Mean average human response delay following escalation. |
| Human reply delay after a supervisor takeover | 18.663 seconds | Mean average human response delay following escalation. |
| Human messages after a technical escalation | 9.966 messages | Mean count of messages sent by the human following escalation. |
| Human messages after an emotional escalation | 8.907 messages | Mean count of messages sent by the human following escalation. |
| Matched repeat contact difference for emotional escalations | 6.0 percentage points | Coefficient 0.060 converted to percentage points; same issue within seven days. |
| Matched rating difference for emotional escalations | -0.928 rating points | Difference in a one-to-five customer rating. |
| Matched rating difference for supervisor takeovers | -0.524 rating points | Difference in a one-to-five customer rating. |
What changes when you compare the figures
| Comparison | Result | Calculation |
|---|---|---|
| Handoffs as a share of supervised chats | 65.001355% | = (4879 + 954 + 1362) / 11069 * 100 |
| Emotional escalation share of handoffs | 13.259208% | = 954 / (4879 + 954 + 1362) * 100 |
| Technical versus emotional human round share gap | 22.1 percentage points | = 65.4 - 43.3 |
| Emotional versus supervisor human reply delay gap | 2.302 seconds | = 20.965 - 18.663 |
| Technical versus emotional human message gap | 1.059 messages | = 9.966 - 8.907 |
| Emotional versus supervisor matched rating gap | 0.404 rating points | = -0.524 - (-0.928) |
Methodology and sources
We read Sections 3.4 and 5.2 and Tables 5–6. Counts come from the supervised AI-eligible subset. For Table 6 the authors record human rounds after escalation and divide by all rounds in each chat, then report group means. We converted the three published proportions to percentages before subtraction. Table 5 uses matched human-only controls and a seven-day window for repeat contacts on the same issue. The contrasts across escalation groups are descriptive.
Download the source data (CSV)
Download the calculations (CSV)
Related reading
AI customer support escalation statistics
AI customer support productivity statistics
AI customer support quality and repeat contact statistics
Peak-season ecommerce support queue statistics by shopper question
Ecommerce support channel statistics for deciding where to staff chat
Sources and definitions
Wang et al., Agentic AI and Human-in-the-Loop Interventions
The source data includes the population, period, denominator and limitations for every figure.