Retail AI support benchmark statistics reveal how many cases require more than answering a question. In Sierra Research’s 114 English retail tasks, 104 specify a state-changing action and 44 specify two or more. Half of the cases require both an order lookup and a write among their reference actions (Sierra Research, Oct 2026).
We count the benchmark’s required tasks by change type, identity check, order references, number of writes and named tools. The counts describe designed test cases. They provide a practical reference for a merchant deciding what to verify before a retail assistant says an order has changed. It also shows why an assistant needs different checks for a confirmed change and a promise to check later.
Key Takeaways
- 104 of 114 English retail benchmark cases expect a write. (Sierra Research, Oct 2026).
- 44 of those benchmark cases expect multiple writes. (Sierra Research, Oct 2026).
- 58.1% of return-expected benchmark cases have repeated order lookups. (Sierra Research, Oct 2026).
- 61.0% of pending-edit benchmark cases expect multiple writes. (Sierra Research, Oct 2026).
- 56.1% of v2.0.0 English retail cases expected an order-details lookup. (Sierra Research, Oct 2026).
- 50.0% of those cases expected the name-and-ZIP lookup. (Sierra Research, Oct 2026).
- 28.9% of English retail cases referenced two distinct explicit order IDs. (Sierra Research, Oct 2026).
- 14.9% referenced three distinct explicit order IDs. (Sierra Research, Oct 2026).
- 50.0% of tasks specify both an order lookup and a state-changing action. (Sierra Research, Oct 2026).
- 41.2% specify a write without a listed order details lookup. (Sierra Research, Oct 2026).
- 56.1% of v2.0.0 tasks expect an order details action. (Sierra Research, Oct 2026).
- 3.5% of benchmark cases specify a human transfer. (Sierra Research, Oct 2026).
How many tested requests require a change to an order or account?
The English retail benchmark expects many cases to edit a record. A correct conversational answer can still fail the evaluation if the underlying order remains in the wrong state.

How common are writes and supporting reads?
Expected calls provide a way to separate retrieval, changes and transfers. The benchmark assesses how a case should be completed, rather than how many customers in a real store request each change.
- Tasks expecting at least one write. 91.2% (104 of 114) in 114 English retail tasks, Sierra Research v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 38.6% (44 of 114) in 114 English retail tasks, Sierra Research v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 56.1% (64 of 114) in 114 English retail tasks, Sierra Research v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 36.8% (42 of 114) in 114 English retail tasks, Sierra Research v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 3.5% (4 of 114) in 114 English retail tasks, Sierra Research v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 57.9% (66 of 114) in 114 English retail tasks, Sierra Research v2.0.0 (Sierra Research, Oct 2026).
How many writes can a single request need?
These groups partition the case set by expected write count. Some cases can require more than one edit even when the customer expresses them in one natural request.
- Tasks with 0 expected writes. 8.8% (10 of 114) in 114 English retail test scenarios, v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 1 expected writes. 52.6% (60 of 114) in 114 English retail test scenarios, v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 2 expected writes. 20.2% (23 of 114) in 114 English retail test scenarios, v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 3 expected writes. 13.2% (15 of 114) in 114 English retail test scenarios, v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 4 expected writes. 4.4% (5 of 114) in 114 English retail test scenarios, v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 5+ expected writes. 0.9% (1 of 114) in 114 English retail test scenarios, v2.0.0 (Sierra Research, Oct 2026).
A helpful evaluation checks both the text and the result in the order record. Include an edit that succeeds, one that is blocked by policy, and a multi-part request with one unavailable step. The confirmation should identify each successful change and what remains to complete.
Which retail requests expose different dependencies?
Returns, exchanges, edits and cancellations can require different records and permissions. Counting within each task category gives a way to check recurring state-changing requirements.

How do return and exchange cases differ?
A return can have the correct items and an unsupported refund instrument, while an exchange can require both eligibility and a replacement option. Check each required relationship in the task’s record.
- Tasks expecting at least one write. 100.0% (31 of 31) in English retail cases with expected returns, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 48.4% (15 of 31) in English retail cases with expected returns, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 64.5% (20 of 31) in English retail cases with expected returns, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 58.1% (18 of 31) in English retail cases with expected returns, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 3.2% (1 of 31) in English retail cases with expected returns, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 64.5% (20 of 31) in English retail cases with expected returns, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 100.0% (29 of 29) in English retail cases with expected exchanges, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 31.0% (9 of 29) in English retail cases with expected exchanges, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 51.7% (15 of 29) in English retail cases with expected exchanges, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 13.8% (4 of 29) in English retail cases with expected exchanges, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 29) in English retail cases with expected exchanges, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 51.7% (15 of 29) in English retail cases with expected exchanges, Sierra v2.0.0 (Sierra Research, Oct 2026).
How much change appears in pending edits and cancellations?
The cases show that a single customer conversation can affect more than one field or order. A narrow edit should preserve other order data and record why an unavailable action could not be completed.
- Tasks expecting at least one write. 100.0% (41 of 41) in English retail cases with expected editing, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 61.0% (25 of 41) in English retail cases with expected editing, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 48.8% (20 of 41) in English retail cases with expected editing, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 36.6% (15 of 41) in English retail cases with expected editing, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 41) in English retail cases with expected editing, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 51.2% (21 of 41) in English retail cases with expected editing, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 100.0% (18 of 18) in English retail cases with expected cancelling, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 72.2% (13 of 18) in English retail cases with expected cancelling, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 50.0% (9 of 18) in English retail cases with expected cancelling, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 44.4% (8 of 18) in English retail cases with expected cancelling, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 18) in English retail cases with expected cancelling, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 55.6% (10 of 18) in English retail cases with expected cancelling, Sierra v2.0.0 (Sierra Research, Oct 2026).
For a merchant, test at least one permitted and disallowed request for each important operation. Confirm the order state after the action and verify the reply against that state. An unprocessed attempt should preserve the original order and say what the shopper can do next.
How often do the cases rely on order or identity lookups?
An assistant needs different evidence for identity, a product and an actual order. Reference action counts help show why a single order lookup can be insufficient for some tasks.

How do the distinct information requirements compare?
An identity lookup establishes which user the proposed action belongs to. A product lookup establishes which item or variant applies. An order lookup concerns its actual status and details.
- Cases with a reference find user id by name zip action. 50.0% (57 of 114) in All English retail test cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Cases with a reference find user id by email action. 9.6% (11 of 114) in All English retail test cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Cases with a reference get user details action. 48.2% (55 of 114) in All English retail test cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Cases with a reference get product details action. 29.8% (34 of 114) in All English retail test cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Cases with a reference get order details action. 56.1% (64 of 114) in All English retail test cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
What else does a lookup-dependent case often need?
Conditioning on one reference tool shows which other actions can coexist with that tool. This can guide QA toward checks that matter to the reply and final transaction state.
- Tasks expecting at least one write. 91.2% (52 of 57) in English retail tasks with find_user_id_by_name_zip reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 22.8% (13 of 57) in English retail tasks with find_user_id_by_name_zip reference action (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 96.5% (55 of 57) in English retail tasks with find_user_id_by_name_zip reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 61.4% (35 of 57) in English retail tasks with find_user_id_by_name_zip reference action (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 1.8% (1 of 57) in English retail tasks with find_user_id_by_name_zip reference action (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 100.0% (57 of 57) in English retail tasks with find_user_id_by_name_zip reference action (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 81.8% (9 of 11) in English retail tasks with find_user_id_by_email reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 45.5% (5 of 11) in English retail tasks with find_user_id_by_email reference action (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 81.8% (9 of 11) in English retail tasks with find_user_id_by_email reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 63.6% (7 of 11) in English retail tasks with find_user_id_by_email reference action (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 18.2% (2 of 11) in English retail tasks with find_user_id_by_email reference action (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 100.0% (11 of 11) in English retail tasks with find_user_id_by_email reference action (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 87.3% (48 of 55) in English retail tasks with get_user_details reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 29.1% (16 of 55) in English retail tasks with get_user_details reference action (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 98.2% (54 of 55) in English retail tasks with get_user_details reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 69.1% (38 of 55) in English retail tasks with get_user_details reference action (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 5.5% (3 of 55) in English retail tasks with get_user_details reference action (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 100.0% (55 of 55) in English retail tasks with get_user_details reference action (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 97.1% (33 of 34) in English retail tasks with get_product_details reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 23.5% (8 of 34) in English retail tasks with get_product_details reference action (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 100.0% (34 of 34) in English retail tasks with get_product_details reference action (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 52.9% (18 of 34) in English retail tasks with get_product_details reference action (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 34) in English retail tasks with get_product_details reference action (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 100.0% (34 of 34) in English retail tasks with get_product_details reference action (Sierra Research, Oct 2026).
If the user has already proved the required identity in the conversation, avoid asking them to send the same details again. If the requested order is ambiguous, resolve the exact record before taking an action. Log what qualified a write alongside the latest successful record.
Related research on Support quality and repeat contacts.
How often do tasks touch several order IDs?
The number of explicit order references shows where partial progress can be hidden by a broad confirmation. A successful update to one order does not complete another order named in the same request.

What does the count show by task?
An explicit ID is a conservative measure of how many objects the trace names. An indirect reference can still identify an order. The counted task need not change every object it reads.
- Tasks with 0 distinct explicit order IDs across expected actions. 2.6% (3 of 114) in English retail test cases, argument-level IDs in v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 1 distinct explicit order IDs across expected actions. 41.2% (47 of 114) in English retail test cases, argument-level IDs in v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 2 distinct explicit order IDs across expected actions. 28.9% (33 of 114) in English retail test cases, argument-level IDs in v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 3 distinct explicit order IDs across expected actions. 14.9% (17 of 114) in English retail test cases, argument-level IDs in v2.0.0 (Sierra Research, Oct 2026).
- Tasks with 4 distinct explicit order IDs across expected actions. 4.4% (5 of 114) in English retail test cases, argument-level IDs in v2.0.0 (Sierra Research, Oct 2026).
How does the dependence change inside multi-order groups?
Keep the subgroup denominator in view. A multi-order case may require verification of both orders before one eligible change, or separate writes and results for each order.
- Tasks expecting at least one write. 0.0% (0 of 3) in English retail cases with 0 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 0.0% (0 of 3) in English retail cases with 0 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 0.0% (0 of 3) in English retail cases with 0 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 0.0% (0 of 3) in English retail cases with 0 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 33.3% (1 of 3) in English retail cases with 0 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 0.0% (0 of 3) in English retail cases with 0 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 93.6% (44 of 47) in English retail cases with 1 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 12.8% (6 of 47) in English retail cases with 1 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 42.6% (20 of 47) in English retail cases with 1 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 0.0% (0 of 47) in English retail cases with 1 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 47) in English retail cases with 1 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 46.8% (22 of 47) in English retail cases with 1 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 90.9% (30 of 33) in English retail cases with 2 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 69.7% (23 of 33) in English retail cases with 2 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 51.5% (17 of 33) in English retail cases with 2 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 45.5% (15 of 33) in English retail cases with 2 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 6.1% (2 of 33) in English retail cases with 2 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 51.5% (17 of 33) in English retail cases with 2 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 100.0% (17 of 17) in English retail cases with 3 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 35.3% (6 of 17) in English retail cases with 3 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 82.4% (14 of 17) in English retail cases with 3 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 82.4% (14 of 17) in English retail cases with 3 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 17) in English retail cases with 3 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 82.4% (14 of 17) in English retail cases with 3 explicit order IDs in reference actions (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 80.0% (4 of 5) in English retail cases with 4 explicit order IDs in reference actions (Sierra Research, Oct 2026).
Test a request that returns two orders when only one is eligible. The correct system should confirm the successful return for the right order, explain why the other was held and leave that record intact. A combined confirmation can hide the state that matters most to a shopper.
What does a complete write and lookup test need to verify?
Looking only at successful reference writes would miss cases where the order must first be inspected, and looking only at successful lookup would miss state changes. Co-occurrence makes the distinction measurable.

What is the link between lookup and write expectations?
A task with a write but no explicit order-details reference can still expect other retrieval or policy checks. The literal filter only separates the named order lookup from record changes.
- Tasks in order-write and order-lookup category both. 50.0% (57 of 114) in All 114 English retail cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks in order-write and order-lookup category write only. 41.2% (47 of 114) in All 114 English retail cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks in order-write and order-lookup category read only. 6.1% (7 of 114) in All 114 English retail cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
- Tasks in order-write and order-lookup category neither. 2.6% (3 of 114) in All 114 English retail cases, Sierra v2.0.0 (Sierra Research, Oct 2026).
Which cases use repeated writes and identity information?
Categories are based on exact write count, and their other requirements use the same membership. Some small groups make a useful test example without supporting a prevalence estimate for retail support.
- Tasks expecting at least one write. 0.0% (0 of 10) in English retail cases with 0 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 0.0% (0 of 10) in English retail cases with 0 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 70.0% (7 of 10) in English retail cases with 0 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 40.0% (4 of 10) in English retail cases with 0 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 30.0% (3 of 10) in English retail cases with 0 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 70.0% (7 of 10) in English retail cases with 0 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 100.0% (60 of 60) in English retail cases with 1 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 0.0% (0 of 60) in English retail cases with 1 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 65.0% (39 of 60) in English retail cases with 1 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 36.7% (22 of 60) in English retail cases with 1 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 1.7% (1 of 60) in English retail cases with 1 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 68.3% (41 of 60) in English retail cases with 1 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 100.0% (23 of 23) in English retail cases with 2 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 100.0% (23 of 23) in English retail cases with 2 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 34.8% (8 of 23) in English retail cases with 2 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 26.1% (6 of 23) in English retail cases with 2 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 23) in English retail cases with 2 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 34.8% (8 of 23) in English retail cases with 2 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 100.0% (15 of 15) in English retail cases with 3 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 100.0% (15 of 15) in English retail cases with 3 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 46.7% (7 of 15) in English retail cases with 3 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two order lookups. 46.7% (7 of 15) in English retail cases with 3 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting a handoff. 0.0% (0 of 15) in English retail cases with 3 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting customer identity lookup. 46.7% (7 of 15) in English retail cases with 3 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least one write. 100.0% (5 of 5) in English retail cases with 4 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting at least two writes. 100.0% (5 of 5) in English retail cases with 4 reference writes, v2.0.0 (Sierra Research, Oct 2026).
- Tasks expecting an order lookup. 60.0% (3 of 5) in English retail cases with 4 reference writes, v2.0.0 (Sierra Research, Oct 2026).
Log the precondition, request result and postcondition for each material change. Check whether the resulting order state matches the reply. If a later step fails, the test should verify that the system describes partial completion and does not change an unrelated field while trying to recover.
Which reference actions belong in a practical evaluation set?
Individual tools provide a concrete starting place for merchant coverage. Map each named benchmark action to a real support decision in the store before using the benchmark mix for an operating plan.

Which read and write operations appear in the file?
Expected read actions identify whether the store can provide the right order, item or customer information. Their frequency helps make a benchmark coverage report meaningful, with separate destinations for source-of-truth checks.
- Cases that expect get order details. 56.1% (64 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect find user id by name zip. 50.0% (57 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect get user details. 48.2% (55 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect get product details. 29.8% (34 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect find user id by email. 9.6% (11 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect calculate. 9.6% (11 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect get item details. 0.9% (1 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
Which state changes and handoff patterns are represented?
A transfer describes a reference expectation in the task, not a staffing requirement. The store’s live issue mix and unresolved cases determine when a human route is needed.
- Cases that expect modify pending order items. 30.7% (35 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect return delivered order items. 27.2% (31 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect exchange delivered order items. 25.4% (29 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect modify pending order address. 17.5% (20 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect cancel pending order. 15.8% (18 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect modify user address. 8.8% (10 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect transfer to human agents. 3.5% (4 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
- Cases that expect modify pending order payment. 0.9% (1 of 114) in All 114 English retail reference cases, v2.0.0 (Sierra Research, Oct 2026).
Build evaluation slices for retrieving an order, allowed changes, disallowed changes, partially successful changes, and escalation. Report whether the shopper got a correct answer and whether the transaction finished in the correct state. If a team has no verified production result, score the interaction and action trail on a representative controlled fixture.
Related research on AI chat handoff and human response.
Methodology and Sources
We inspected all 114 English retail scenarios in Sierra Research’s tagged benchmark. We count unique task IDs and the expected named actions listed for each task. Return, exchange, cancel and modify prefixes qualify as writes. We record order details lookups, identity lookup and expected human transfers separately. A distinct order reference counts an explicit order ID in any reference action argument and can miss IDs recorded in another form. Subsets can overlap. The figures describe simulated benchmark design and its reference criteria. They do not measure model success, deployed support volumes, escalation rates or the effect of a merchant automation.
Glossary
- Reference action: An expected tool action the benchmark lists for a task.
- Write: An expected change to a retail record.
- Lookup: Retrieval of current information without the same recorded state change.
- Distinct order ID: An explicit unique order identifier found in reference arguments.
- Transfer: Expected handoff to a human agent in the task.
- Task set: A versioned collection of synthetic evaluation cases.
- Postcondition: The state that should be verified after an attempted action.
- Partial completion: Some requested changes finish while others remain open.