How AI Shopping Agents Choose Products

13 min read

TL;DR Tap to expand the short version

An AI agent treats your review section as two separate things: an aggregate it does arithmetic on (star rating, review count) and a block of text it reads as language. Almost all review advice addresses only the first.
In randomised trials on a controlled storefront, where a product sat on the page moved selection more than its review signals did. A top row slot carried a coefficient of +1.22 for one model. An “Overall Pick” badge carried up to +1.90.
Instructions planted inside review text hijacked deployable shopping agents in 41.67% to 68.16% of runs across 3,168 adversarial trials. One template that simply skewed the agent’s product assessment worked 100% of the time on two of four systems.
A star rating that contradicts a suspicious review is a real, measurable defence on some models and no defence at all on others.
There is still no good study of how agents read ordinary review text when nobody is attacking them. I ran a small one. Two of three frontier models downgraded a product by quoting Amazon’s AI-generated review summary rather than any actual reviewer, and the sentiment they cited appears in none of the reviews they were shown.

Retailers have spent years and real money lifting product ratings. Their competitors did the same.

The three tablets I tested last week cost $549, $859 and an unknown amount because one listing showed no price. All carried a 4.7 rating.

A tied rating cannot choose a winner. An AI shopping agent must use something else, and retailers have little evidence about what that is.

In July 2026, I gave three live Amazon tablet listings to GPT-5 in ChatGPT, Claude Sonnet 5 and Gemini 3.6 Thinking. The brief asked for the best overall value for daily use, media, light work and occasional notes. Each model saw listing data first, then the same data with customer reviews. Two models marked the iPad down for mixed battery sentiment. Yet all five battery comments in the supplied reviews were positive.

The complaint appeared elsewhere on the page, inside Amazon’s AI-generated “Customers say” box. That summary draws on thousands of reviews the models did not receive. Both models treated a machine-written summary as customer sentiment.

Six answers cannot establish a pattern. They did, however, expose a gap in the research. Recent studies show how agents respond to ratings, placement and hostile review text, but say far less about how they read ordinary customer reviews.

The fake shop

Researchers who want to observe an AI agent buying products first need a controlled shop.

A team from Columbia Business School, Yale and MyCustomAI built one. Their storefront varied product position, price, rating, review count and badges across eight categories, with 1,000 trials per category.

In staplers, Amazon Basics won repeatedly while a brand called Arrow was never selected.

The study is called ACES, it was revised in December 2025, and it is now published at the ACM Web 2026.

Shelf position wins

Moving a product from the bottom of the grid to the top row made one agent roughly five times more likely to buy it. The product, price and reviews did not change.

An “Overall Pick” badge increased selection by a Gemini-powered agent. A “Sponsored” tag reduced selection across every model tested.

Effect on the odds an agent picks the product. Bars show magnitude; orange pushed the agent away.

Top row of the grid

Claude Sonnet 4 +1.22

GPT-4.1 +1.05

Gemini 2.5 Flash +0.34

Overall Pick badge

Claude Sonnet 4 +1.06

GPT-4.1 +0.80

Gemini 2.5 Flash +1.90

Reviews as instructions

Review text creates a different risk. A language model receives user instructions and webpage content in the same input, and may treat a sentence written by a reviewer as an instruction.

StakeBench tested that risk through 3,168 runs with two downloadable shopping agents and frontier models. Researchers changed only review text, ratings and seller metadata.

Attack success rate across 3,168 runs on two deployable shopping agents.

NanoBrowser · GPT-5 41.7%

BrowserUse · GPT-5 53.0%

BrowserUse · Gemini 2.5 Flash 59.2%

NanoBrowser · Gemini 2.5 Flash 68.2%

Source: Wang et al., StakeBench, Nanyang Technological University, ST Engineering, IBM Research and UIUC, arXiv:2606.13385, June 2026. Labels assigned by an automated judge with 90.63% agreement against human annotation.

A quiet instruction designed to change the agent’s opinion of a product worked in every run against both GPT-5 setups and in 94.4% of one Gemini pairing.

Instructions close to the shopper’s request were most effective. Steering the agent towards a competing product in the same category succeeded in roughly 71% to 79% of runs. Unrelated instructions succeeded far less often and produced more obvious odd behaviour.

The researchers call the quiet attack stealthy parasitism. The recommendation still looks sensible, so the shopper and competing retailer may never see the manipulation.

Conflicting signals

The team then created a conflict between a malicious review and its star rating.

For GPT-5, attack success fell from 55.6% to 19.4%. The model noticed the mismatch and trusted the review less.

Gemini 2.5 Flash showed almost no change, moving from 83.3% to 86.1%.

A small side experiment altered only a product image. Its selection rate rose from 10% to 76.7%, showing that text is not the only exposed channel.

Five page checks

The five checks below translate the findings into a product-page review that can be run in an afternoon.

Step	What to do	Why it matters
1. Read the aggregate first	Write down your star rating and review count exactly as they render. Compare against the three products that outrank you.	These are the only review signals that reach an agent working from a results grid.
2. Check the frame	Note your grid position and any badges on the listing, including sponsored labels.	Placement carried more weight than review signals, and a sponsored tag counted against every model tested.
3. Test the text channel	Read your ten most recent reviews as if they were commands. Flag anything phrased as an instruction, a system notice, or an authority claim.	Agents cannot separate instructions from content, and the subtle attacks are the effective ones.
4. Check cross-cue consistency	Look for reviews whose star rating disagrees with their text. Investigate rather than delete.	Contradiction cut attack success by two thirds on GPT-5. It did nothing on Gemini.
5. Re-run after model releases	Repeat steps 1 to 4 whenever a major model ships. Log what changed.	Model updates behave like demand shocks, and the direction is not predictable.

Model updates can also reorder the shelf. A Fitbit Inspire was chosen by a Claude-powered agent 45% of the time in August 2025 and 77% in December. Over the same period, its selection by an OpenAI model fell from 25% to 6%.

One model update, and model shelf shifts

Share of trials in which the agent chose the Fitbit Inspire. Same product, same listing, same page.

Anthropic agent

Sonnet 4 · Aug 2025 45%

Opus 4.5 · Dec 2025 77%

OpenAI agent

GPT-4.1 · Aug 2025 25%

GPT-5.1 · Dec 2025 6%

Source: Allouah, Besbes, Figueroa, Kanoria and Kumar, ACES, arXiv:2508.02630v3, December 2025.

The product and price stayed the same while model releases shifted its share of agent selections. In phone cases, one brand fell from 62.6% with GPT-4.1 to 5% with GPT-5.1.

Retailers therefore need scheduled testing after major model releases instead of assuming yesterday’s agent behaviour still holds.

From lab to web

Real-world prompt injection remains cruder than the laboratory results.

Palo Alto researchers examined prompt injection found in live web telemetry in March 2026. They found opportunistic attacks that were generally less capable than benchmark demonstrations.

The most common attacker intent by far was irrelevant output, at 28.6%. Jokes, anti-scraping notices, cabbage.
85.2% of the jailbreak attempts were plain social engineering. No clever encoding, no adversarial optimisation.
75.8% of affected pages carried a single injected prompt.
They logged what they believe is the first real case of an AI ad review system being bypassed this way, in December 2025, and noted they have not confirmed any successful attack against a deployed ad-checking agent.

The evidence supports a measured response. The vulnerability is well documented, while exploitation in the wild remains limited and unsophisticated.

Any urgent commercial claim should explain which threat it addresses and what evidence shows that threat exists outside a lab.

Review law applies

Review manipulation was illegal before shopping agents appeared. The FTC’s review rule took effect on 21 October 2024 and bans buying, selling and distributing fake reviews, including AI-generated ones. Violations can attract civil penalties.

The most-cited estimate is that around 4% of online reviews are fake, influencing $152 billion in global spending. That five-year-old estimate relies on platform self-reporting. Treat 4% as a floor of unknown accuracy rather than a fact.

The reader has changed. A fake review may now influence software before a person sees it.

Ordinary reviews untested

Together, the studies leave the most ordinary case unanswered.

ACES estimates how agents respond to product-grid signals but includes no review reading. StakeBench studies review text only when it contains an attack. Neither measures an agent reading a normal set of genuine reviews at scale.

Claims that detailed reviews improve agent recommendations still rely mainly on commercial blog posts. Published research has not established the effect.

The tablet test

That gap is why I ran the tablet test described at the start.

Method

Samsung Galaxy Tab S10 FE, 128GB, $549.99, rated 4.7 from 940 ratings
Apple iPad Air 13-inch (M2), 512GB, $859.00, rated 4.7 from 1,200 ratings
Samsung Galaxy Tab S10+ 12.4 inch, 256GB, no price shown on the listing, rated 4.7 from 757 ratings

Condition A supplied the product title, specifications, price where available, star rating and review count.

Condition B added the top US reviews, Amazon’s “Customers say” summary and its sentiment tags. Amazon generates that summary from a wider review set than the visible comments.

Each model gave a recommendation, confidence rating and reasons. I removed the S10 FE’s Sponsored label to avoid adding another variable.

What happened

Model	Condition A	Condition B	Stated confidence
GPT-5	Galaxy Tab S10+	Galaxy Tab S10 FE	Medium to High
Claude Sonnet 5	Galaxy Tab S10 FE	Galaxy Tab S10 FE	Medium-High to High
Gemini 3.6 Thinking	Galaxy Tab S10 FE	Galaxy Tab S10 FE	High to High

Only GPT-5 changed its recommendation. It first chose the S10+ despite noting that the missing price prevented a value comparison. After seeing reviews, it moved to the cheaper S10 FE because owners described the larger tablet as heavy. All three models then recommended the same product, and two reported higher confidence.

Gemini described the S10+ as “kinda pricey” and “way overkill.” The first phrase came from a positive review that continued, “but I promise the price deserves it.” The model retained the concession and lost the verdict.

Reviews also supplied facts missing from the specifications. Gemini credited the S10 FE with microSD expansion after a reviewer mentioned adding a card for work.

Test limits

One run per model and condition cannot separate a real effect from sampling noise. Product order was fixed, the S10+ had no listed price, and both factors could have influenced the result.

The test shows that models used the summary layer, but it cannot estimate how much that layer changes recommendations.

Where to start

Start with one product, run the five checks and record the result. Repeat after major model releases, when the evidence suggests product positions can move.

For the wider context on how machine readers are reshaping discovery, our piece on agent advertising covers the same shift from the media-buying side. On the retail side, agentic checkout is already live, even though almost nobody is using it yet.

FAQ

Do AI agents analyze the actual text of product reviews or only the star ratings?

Agents use the information available to them. On results pages that may be ratings and counts. On a product page it can include review text and platform-generated summaries.

Is it possible for a user to manipulate an AI agent’s behavior through review text?

StakeBench shows that instructions inside reviews can influence agents in controlled tests. Evidence of comparable real-world exploitation remains limited.

Does a higher volume of reviews guarantee that an AI will recommend my product?

Ratings and review counts can help, but product position and platform badges sometimes have a larger effect.

What are the legal implications of using AI to generate fake reviews in the United States?

The FTC rule under 16 CFR Part 465 bans creating, buying, selling and distributing fake reviews. It took effect in October 2024 and allows civil penalties.

How can businesses defend against malicious attempts to manipulate AI agents via reviews?

A mismatch between a rating and malicious review text reduced attack success for GPT-5 in one test but not for Gemini. It is a useful signal, not a universal defence.

False order confirmations, on what happens when the assistant misstates your customer’s cart.
Agentic checkout gap, on the adoption gap between merchant readiness and actual consumer use.
AI recommendation signals, on the reputation signals that decide whether an agent recommends you at all.
How to Advertise, on what happens to paid media once bots account for the majority of web traffic.

Spotted an error, have a correction, or want to pitch a story? Contact [email protected].

How AI Shopping Agents Read Product Ratings and Reviews

Table of Contents

The fake shop

Shelf position wins

Reviews as instructions

Conflicting signals

Five page checks

From lab to web

Review law applies

Ordinary reviews untested

The tablet test

Method

What happened

Test limits

Where to start

FAQ

Do AI agents analyze the actual text of product reviews or only the star ratings?

Is it possible for a user to manipulate an AI agent’s behavior through review text?

Does a higher volume of reviews guarantee that an AI will recommend my product?

What are the legal implications of using AI to generate fake reviews in the United States?

How can businesses defend against malicious attempts to manipulate AI agents via reviews?

Like this:

How AI Shopping Agents Read Product Ratings and Reviews

Table of Contents

The fake shop

Shelf position wins

Reviews as instructions

Conflicting signals

Five page checks

From lab to web

Review law applies

Ordinary reviews untested

The tablet test

Method

What happened

Test limits

Where to start

FAQ

Do AI agents analyze the actual text of product reviews or only the star ratings?

Is it possible for a user to manipulate an AI agent’s behavior through review text?

Does a higher volume of reviews guarantee that an AI will recommend my product?

What are the legal implications of using AI to generate fake reviews in the United States?

How can businesses defend against malicious attempts to manipulate AI agents via reviews?

Related reading

Share this:

Like this:

Discover more from Industry Contents