Journey Hallucinations: When AI Agents Invent Ecommerce Orders

22 min read

Editor’s note: This piece is built from primary sources, incl. the UMAP paper written by employees of companies that sell a product based on its findings. We say so wherever its numbers appear, and we have reproduced the limitations its own authors declared.

TL;DR Tap to expand the short version

A UMAP paper found that AI commerce agents claimed successful order completion, even though the event log was empty. The authors call these journey hallucinations.
Across 90 real sessions and four foundation models, ungrounded prompting produced these false claims at rates up to 8.5%. GPT-4o was the worst of the four.
Independent work points the same way. On τ-bench, a GPT-4o retail agent scoring above 60% on a single attempt fell below 25% when asked to get the same task right eight times running.
To prevent this, run the five checks in the verification checks below before an agent goes near a cart, and ask any vendor selling you a fix the five questions in the section after it. Most of the work is logging you already have.

Explaining the Journey Hallucination

Most discussion of AI hallucination concerns invented facts: citations, statistics or court cases. Transactional agents introduce a different error. Researchers from Rezolve AI Labs and the University of Sarajevo call it a journey hallucination: a mismatch between what an agent says happened during a shopping session and what the event log records.

In practice it sounds like this. In this real example from their user study, where the agent was describing a session in which the shopper had added items, removed some, completed a purchase, and still had three items in the cart:

“The cart is currently empty, the checkout process has started, and an order has been completed.”

Two of the three statements were false, and no control-group participant judged the response accurate. Transactional errors matter because downstream systems act on them. An invented cart or order can trigger recovery messages, post-purchase communication or support workflows for events that never occurred.

Anyone who has run lifecycle marketing knows what those emails do to a customer’s opinion of you. The paper further narrows the problem to four claims. They were chosen because each one can be recovered from an event log, each admits a straight true or false comparison, and each maps directly onto something automable that fires.

The four claims you should verify

Is the cart empty?

Wrong answer fires cart recovery at someone who never added anything.

What is in the cart?

Wrong answer produces cross-sell on items that are not there, and pricing conversations about the wrong basket.

Has checkout started?

Wrong answer sends abandonment flows and payment troubleshooting to people who never reached the form.

Is the order complete?

The expensive one. Wrong answer triggers post-purchase messaging, delivery expectations and refund requests for an order that does not exist.

Source: Carvalho, Krivic, Tang and Ahmad, ACM UMAP 2026. Consequences are ours.

What event logs show

The team took 90 real ecommerce sessions containing 2,000 events, drawn from two merchants on a shared conversational commerce platform: a fashion retailer and a luxury goods reseller. Sessions averaged 22 events. Every session included at least one cart action.

They then replayed each session step by step, asked four foundation models to state what was going on in the cart, and compared the answers to the log. All four ran at temperature zero on identical prompts: GPT-4o, DeepSeek-V3, Kimi-K2-Instruct and Qwen3-32B.

The most important metric to look for is the false positive rate, meaning the agent claimed forward progress that never happened. That is the harmful subclass, because those are the claims that trigger action.

False claims of forward progress

Share of interactions where the agent asserted a funnel event the log does not support. Lower is better.

GPT-4o

8.5%

Kimi-K2

4.1%

Qwen3-32B

3.1%

DeepSeek-V3

2.4%

The same four models, once the event log was replayed into the prompt

GPT-4o

0.4%

Kimi-K2

0.2%

Qwen3-32B

0.0%

DeepSeek-V3

0.0%

Source: ACM UMAP 2026, 90 sessions across two merchants. Figures aggregate cart emptiness and checkout status claims. Bars scaled to the 8.5% maximum.

The full picture, including how often each model was willing to answer at all:

Model	Coverage	Accuracy	False positives	False negatives
GPT-4o	62.3%	86.2%	8.5%	5.3%
Kimi-K2-Instruct	99.4%	93.5%	4.1%	2.5%
Qwen3-32B	63.9%	96.6%	3.1%	0.3%
DeepSeek-V3	81.2%	92.1%	2.4%	5.5%

Ungrounded prompting. Coverage is the share of questions answered rather than skipped. Source: ACM UMAP 2026, Table 2.

GPT-4o produced the lowest accuracy and coverage and the highest rate of unsupported forward progress. Qwen3-32B, a smaller open-weight model, reached 96.6% accuracy. GPT-4o’s 8.5% false-positive rate is roughly one interaction in twelve, material at production volume.

Why hedging fails

A common safeguard is to tell the model to admit uncertainty. The researchers tested that condition. Qwen declined roughly 85% of questions, making the assistant largely unusable. Kimi answered only about a quarter and its false-positive rate increased from 4.1% to 8.1%.

Caution did not make it honest. It made it quieter and, on the occasions it did speak, less reliable than before. The paper’s phrasing is that conservative prompting reduces errors mainly through abstention, producing agents that are safe but rarely helpful. Which is a polite way of saying it does not work.

Repeated tests expose failures

There is a second body of evidence that arrives at the same place from a different direction, and it explains why pilots look fine and production does not.

τ-bench, built by researchers at Sierra, tests agents on realistic retail and airline customer service tasks with tools, policies and a simulated user. It introduced a metric called pass^k: the share of tasks the agent gets right on all k attempts, rather than at least one.

Test condition	Success rate
One attempt	Above 60%
Eight consecutive attempts	Below 25%

GPT-4o function-calling agent, τ-bench retail domain. Source: arXiv 2406.12045.

The practical translation is that a demo proves capability and tells you nothing about reliability. Your pilot runs each scenario once. Production runs it several thousand times a week, which is precisely the condition under which the number collapses.

Liability and cost

In November 2022, Air Canada’s chatbot told Jake Moffatt that he could apply retrospectively for a bereavement fare. He booked at full price, travelled to his grandmother’s funeral and was refused because the airline’s published policy said the opposite.

Air Canada’s defence was that the chatbot was responsible for its own statements. On 14 February 2024 the British Columbia Civil Resolution Tribunal rejected the defence, calling it a remarkable submission and ruling that the airline was responsible for all information on its website regardless of whether it came from a static page or a chatbot. Moffatt was awarded C$812.02.

Eight hundred dollars is nothing. The findings are everything, and three of them should shape how you deploy:

The tribunal found a duty of care to take reasonable steps to ensure the bot’s representations were accurate.
Having the correct information elsewhere on the site was not a defence. Air Canada could not explain why the bereavement page was inherently more trustworthy than the bot.
The customer’s reliance on the bot was found to be reasonable.

It is a small tribunal in one Canadian province and it binds nobody else. It is also the closest thing anyone has to a precedent, and it was decided before agents could touch a cart.

Now map the four claim types onto that. An agent telling a customer their order is complete when no order exists is a materially different problem from a bot misdescribing a fare policy. The customer has a screenshot. Your order system has nothing.

How one false claim becomes a cost line

STEP 1

Agent states an order is complete. Log shows no order.

STEP 2

Customer relies on it. Screenshots it. Waits for delivery.

STEP 3

Nothing arrives. Support contact opens. The agent has no record to work from.

STEP 4

Goodwill refund, or a chargeback you cannot contest because there is no transaction.

STEP 5

Review, social post, or a negligent misrepresentation claim on the Moffatt reasoning.

Sizing it for your own business

Published rates are useful for direction and useless for a budget conversation. Here is the arithmetic to run on your own numbers instead. It takes an afternoon and it is the only version anyone in finance will accept.

Two of those inputs you already have from your helpdesk. The third, your own false positive rate, is the number this whole article exists to make you measure. The 90 day plan below sets out how to get it.

Do not model brand damage. It is real and you cannot defend an estimate of it, so leave it out and say you have left it out.

Errors beyond checkout

Transactional state is unusually clear because it can be checked against a log. A related problem appears in generated reviews and summaries. UC San Diego researchers reported that LLM summaries changed source sentiment in 26.42% of cases; questions after the model’s knowledge cutoff produced hallucinations in 60.33%.

The part worrying retailers is the behavioural experiment attached to it. Seventy participants were shown either machine written summaries of product reviews or the original reviews. Products included headlamps, radios and headsets.

A 32-point swing in purchase intent, produced by nothing except a summarisation layer. In the short term that reads like a conversion win. Then the returns arrive, because a chunk of those buyers were sold on a product the actual reviewers were lukewarm about.

The researchers tested 18 mitigation strategies. bias remained across models and tasks.

Five checks before cart access

Call this the verification checks, because every check comes back to the same question: can you prove what the agent said against a record of what happened?

Where verification sits in the request path

1. Event log
viewProduct, addToCart, removeFromCart, checkoutStarted, purchase

2. Replay
Fold events in order into a deterministic state object

3. Prompt
State object goes into context alongside the conversation

4. Agent
Returns a structured claim object plus the customer facing message

5. Verify
Compare claims to state. Pass, or regenerate before sending

If step 5 runs as a nightly batch job instead of in the serving path, the customer has already read the false claim. You have built a report, not a control.

1. Write down the claims that can hurt you

You cannot verify an agent’s output in general. You can verify a short list of specific assertions, and a short list turns out to be enough. Start with the paper’s four, then add any claim that satisfies all three of these conditions. It must be recoverable from an event log. It must admit a straight true or false comparison. It must trigger something automatic.

What that looks like across different models of business:

If you sell	Add these claims
Fashion and apparel	Size reserved, return window open, exchange already raised
Subscription	Plan tier, renewal date, pause active, trial consumed
Grocery and delivery	Slot booked, substitution accepted, order past amendment cutoff
Marketplace	Which seller fulfils, dispatch confirmed, dispute already open
High value or regulated	Identity check passed, finance application status, age verification

Anything that fails one of the three conditions stays off the list. Preference, sentiment and intent are not verifiable against a log, and pretending otherwise gives you a control that quietly does nothing.

2. Make the log the source of truth, not the model

Asking a model to infer cart state from a compressed conversation history is asking it to guess. The event stream already knows the answer. Fold the session’s events in order into a state object, then put that object in the prompt. This is the entire mechanism behind the accuracy jump above. The pattern is a reducer, and it is deliberately boring:

			
state = { cart: {}, checkout_started: false, order_completed: false }
for event in session_events_sorted_by_server_timestamp:
    if event.type == \"addToCart\":       state.cart[event.sku] += event.qty
    if event.type == \"removeFromCart\":  state.cart[event.sku] -= event.qty
    if event.type == \"checkoutStarted\": state.checkout_started = true
    if event.type == \"purchase\":        state.order_completed  = true
# state is now the only thing allowed to answer
# \"what is in the cart\" for the rest of this turn

		

Four things break this in production and all four should be checked:

Out of order events. Sort by server timestamp rather than client. Mobile clocks drift.
Duplicate events. Retries and double taps inflate quantities. Make the fold idempotent on event ID.
Cross device sessions. A cart built on mobile and opened on desktop is one journey and two session IDs. Fold on the customer wherever you have identity.
Silent schema drift. A renamed event type will not error, it will quietly produce an empty cart. Alert on state objects that come back empty when the session length says they should not be.

3. Check claims before the response ships, not after

A violation caught after the customer has read “your order is complete” is not caught. It is logged. Have the agent return its state claims as a structured object alongside the natural language reply. Keeping the machine readable claim separate from the customer facing text is the trick that makes this work, because you can verify the first without flattening the tone of the second.

			
{
  \"claims\": {
    \"cart_empty\":       false,
    \"cart_items\":       [ {\"sku\": \"A-112\", \"qty\": 2} ],
    \"checkout_started\": true,
    \"order_completed\":  false
  },
  \"message\": \"You\'ve got two of the linen shirts in your basket and
              you\'re partway through checkout. Want me to carry on?\"
}
# verify claims against replayed state
# any mismatch on a positive assertion  ->  block and regenerate

		

Decide up front what happens on a violation, because retrying forever is not an answer:

Regenerate once with the verified state repeated in the prompt and the violated claim named explicitly.
Fall back to a deterministic sentence if the retry fails. A templated “your basket has 2 items and you have not completed checkout” is dull and correct, which beats fluent and wrong.
Escalate to a human for any violation on order completion, because that is the claim with a refund attached.
Log every violation with the session ID, so the rate becomes a metric rather than an anecdote.

Budget for the latency. A replay and four boolean comparisons are cheap, but a regeneration is a second model call, so size the retry path against your p95 rather than your average.

4. Report false positives separately from accuracy

Overall accuracy hides commercially different errors. Missing a real cart can lose an upsell; inventing one can produce support costs or a chargeback. Track unsupported positive claims separately from missed positives. GPT-4o’s 86.2% accuracy obscures its 8.5% rate of asserting events that did not happen.

A scorecard you can lift straight into a spreadsheet:

Metric	What it means	Why it is separate
Coverage	Share of turns where the agent stated transaction state at all	High accuracy at 20% coverage is an agent that refuses to work
Unsupported positives	Claimed a funnel event the log does not support	The line that generates contacts, refunds and legal exposure
Missed positives	Failed to recognise a real cart, checkout or order	Costs revenue rather than trust. Different owner, different fix
Order-claim violations	Unsupported positives specifically on order completion	Highest severity. Report at zero, not as a percentage
All-eight pass rate	Share of scenarios correct on eight consecutive runs	The only number that predicts production behaviour

5. Test the same task eight times, not once

Reliability decays with repetition in a way single run testing cannot see, and repetition is exactly what a deployed agent faces. Borrow pass^k from τ-bench and build a small harness. Twenty scenarios is enough if they are the right twenty. Pick them from your actual contact reasons rather than from imagination, and make sure the set covers:

An empty session where nothing has happened yet
Add then remove, ending empty. This is the one that catches most journey hallucinations
A completed purchase followed by new browsing in the same session
Checkout started and abandoned
Two simultaneous errors, which the paper found produced the largest gap between verified and unverified responses
A correct baseline, so you can tell whether verification is helping or just changing the tone

Report the all-eight number to whoever signs off the deployment. Report the single-run number nowhere.

Five questions to ask any vendor selling you a fix

Several companies now sell verification layers for exactly this problem, including the one whose researchers wrote the paper. Some of them are good. The questions below separate them, and they work regardless of whose logo is on the deck.

Ask	Why	What a good answer sounds like
What is your false positive rate on unsupported positive assertions?	Accuracy alone hides the expensive errors. Anyone quoting only a headline accuracy figure has not measured the thing that costs you money	A number, split out from overall accuracy, with the sample size attached
What did you compare against?	Beating an ungrounded prompt is not the same as beating your current stack. The UMAP paper is explicit that its baselines were prompt configurations, not production systems	A named comparison, ideally against a session memory system rather than a naive baseline
Does the verifier run in the serving path or after the fact?	A check that runs nightly produces a report. Only an inline check prevents the customer reading the false claim	Inline, with a stated latency budget and a defined fallback
What happens when a check fails?	Blocking without a fallback turns a wrong answer into no answer, which is the abstention trap in a different costume	Regenerate, then a deterministic fallback, then escalation, with thresholds you control
What do you need from our event schema, and what breaks if a field is missing?	The whole approach depends on structured traces. The paper lists that dependency as its first limitation	A specific event list, and an honest answer about degradation rather than a claim that it always works

One more thing worth doing before any of that. Run the first two checks yourself. Not because you will necessarily build it in house, but because you cannot evaluate a proposal for a problem you have not yet measured.

Study limitations

The UMAP paper was written by employees of Rezolve Ai, a company that sells a verification product called TraceWare, and the paper is the scientific case for the approach that product implements. That does not make the numbers wrong. It does mean the caveats deserve as much space as the headline, and to the authors’ credit most of them are declared in the paper itself.

It is a Late Breaking Results paper, the short format at UMAP, rather than a full track submission.
The sample is small. Ninety sessions from two merchants. The authors say so and call for a larger study.
The comparison is not against real products. All three modes tested are prompt configurations of the same framework. The paper states explicitly that they are not a head to head against production middleware or session memory systems. So the result shows trace grounding beats no trace grounding, not that it beats whatever your platform already does.
The user study is 42 people, recruited from the organisation and personal networks, without counterbalancing. The authors describe it as a convenience sample.
It cannot catch subtle errors. The method only detects claims that violate an explicit trace constraint. An agent that is misleading without contradicting the log passes every check.

And then there is scenario S2, which is the most interesting result in the paper and cuts directly against the product it supports.

In S2 the unverified response contained one subtle error about checkout status, wrapped in an otherwise accurate summary. The verified, factually correct response was judged correct by 29% of participants. The wrong one was judged correct by 71%.

People preferred the confident inaccurate answer. The authors’ conclusion is that framing can override accuracy when errors are subtle, and that verification needs pairing with response generation that knows which verified facts to surface and how. Correctness alone does not buy you trust.

A vendor could easily have left that table out.

A 90-day test

You do not need a programme for this. You need one number you almost certainly do not have.

Weeks 1 to 2. Get your own 8.5%. Pull fifty recent sessions where your assistant said anything about a cart, an order or a delivery. Pull the matching event logs. Count how many times the assistant asserted something the log does not support, splitting unsupported positives from missed positives. That percentage, measured on your own traffic, is worth more to your board than any benchmark in this article.
Weeks 3 to 6. Build the replay. Steps one and two of the verification checks. A reducer over your existing event stream, wired into the agent’s context. Most teams find the events already exist and the work is plumbing rather than instrumentation.
Weeks 7 to 10. Put the check in the serving path. Structured claims, inline verification, a defined fallback, violations logged with session IDs. Start in shadow mode if you need to, but set a date to turn on blocking, because shadow mode that never ends is a report.
Weeks 11 to 12. Re-measure and run the harness. Same fifty sessions, same method, report the delta. Then run your twenty scenarios eight times each and take the all-eight number to whoever owns the deployment decision.

If your week two number comes back at zero, you are already grounding properly and none of the rest applies. If it comes back anywhere near one in twelve, you have a support cost and a liability exposure that nobody has budgeted for. It is worth doing before you switch on checkout inside an assistant rather than after.

We looked at where that market actually stands in our piece on agentic checkout.

Sources and method

Source	What we took from it	Independence
UMAP study	The concept, the four invariants, all model level figures, the user study and the S2 reversal	Peer reviewed, Late Breaking Results track. Authors employed by a vendor selling a product based on it
IJCNLP study	26.42% sentiment change, 60.33% hallucination on post-cutoff questions, the 84% versus 52% purchase intent split	Academic, no commercial interest identified
τ-bench	pass^k, and the fall from above 60% to below 25% across eight attempts	Published by researchers at Sierra, which sells agents. Method and data are public
Air Canada case	Liability finding, duty of care, the damages figure	Tribunal decision, publicly reported

This article contains no first party data. Where we have used another organisation’s numbers, the sample size and the date are in the text so you can judge them yourself.

FAQ

What is a journey hallucination?

A journey hallucination occurs when an agent reports a cart, checkout or order event that the execution log does not support. Unlike an ordinary factual error, the claim can trigger automated commercial workflows.

How often do AI agents get transaction state wrong?

In the UMAP study, ungrounded prompting produced unsupported claims about forward progress at rates between 2.4% and 8.5% depending on the model, across 90 real sessions from two merchants. GPT-4o was highest at 8.5%, roughly one interaction in twelve. Those figures come from one study with a small sample, so treat them as an order of magnitude rather than a benchmark, and measure your own.

Is a merchant liable for what its AI agent tells a customer?

In 2024, the British Columbia Civil Resolution Tribunal held Air Canada responsible for incorrect information supplied by its chatbot. The decision is limited to one provincial tribunal and should not be treated as general legal advice.

Can I fix this by telling the model to admit uncertainty?

The study tested that directly and it performed badly. Instructing models to answer unknown when unsure dropped coverage to between 15.3% and 26.7%, meaning the agent declined to answer most of the time. For one model the false positive rate rose from 4.1% to 8.1% under the cautious instruction. The researchers concluded that this approach reduces errors mainly through abstention, producing agents that are safe but rarely useful.

Do I need to buy a product to solve this?

Not necessarily. The method reconstructs transaction state from existing event logs, supplies that state to the model and verifies its claims before delivery. Commercial products implement the pattern, but the underlying control is architectural.

Will a bigger or newer model solve it?

Not by itself. GPT-4o had the study’s highest false-positive rate. Once the event log was supplied, all four tested models reached between 99.5% and 100% accuracy, suggesting that grounding architecture mattered more than model size.

What is pass^k and why should I test eight times?

pass^k measures tasks completed correctly on every repeated attempt. In τ-bench retail testing, a GPT-4o agent scoring above 60% once fell below 25% when success was required across eight runs. Single-run pilots therefore overstate reliability.

Does this affect checkout inside ChatGPT or Gemini?

The study examined agents on merchants’ own conversational platforms, not third-party assistant checkout. The underlying risk still applies, but merchants control logs and verification on their own systems more directly.

Agentic checkout, on why discovery inside assistants is outrunning the buy button.
AI product ratings, on what the model reads on a product page before it ever recommends you.
Agent advertising, on what happens to paid media once bots account for the majority of web traffic.
Brand protection, on keeping listings accurate once you sell in more than one place.

Spotted an error, have a correction, or want to pitch a story? Contact [email protected].

Why AI Agents Produce False Order Confirmations

Table of Contents

Explaining the Journey Hallucination

What event logs show

Why hedging fails

Repeated tests expose failures

Liability and cost

Sizing it for your own business

Errors beyond checkout

Five checks before cart access

1. Write down the claims that can hurt you

2. Make the log the source of truth, not the model

3. Check claims before the response ships, not after

4. Report false positives separately from accuracy

5. Test the same task eight times, not once

Five questions to ask any vendor selling you a fix

Study limitations

A 90-day test

Sources and method

FAQ

What is a journey hallucination?

How often do AI agents get transaction state wrong?

Is a merchant liable for what its AI agent tells a customer?

Can I fix this by telling the model to admit uncertainty?

Do I need to buy a product to solve this?

Will a bigger or newer model solve it?

What is pass^k and why should I test eight times?

Does this affect checkout inside ChatGPT or Gemini?

Like this:

Why AI Agents Produce False Order Confirmations

Table of Contents

Explaining the Journey Hallucination

What event logs show

Why hedging fails

Repeated tests expose failures

Liability and cost

Sizing it for your own business

Errors beyond checkout

Five checks before cart access

1. Write down the claims that can hurt you

2. Make the log the source of truth, not the model

3. Check claims before the response ships, not after

4. Report false positives separately from accuracy

5. Test the same task eight times, not once

Five questions to ask any vendor selling you a fix

Study limitations

A 90-day test

Sources and method

FAQ

What is a journey hallucination?

How often do AI agents get transaction state wrong?

Is a merchant liable for what its AI agent tells a customer?

Can I fix this by telling the model to admit uncertainty?

Do I need to buy a product to solve this?

Will a bigger or newer model solve it?

What is pass^k and why should I test eight times?

Does this affect checkout inside ChatGPT or Gemini?

Related reading

Share this:

Like this:

Discover more from Industry Contents