Why a family jewellery store needed five tests to trust AI

Beating a hand tuned setup is a much harder job than beating no setup at all. Witly, the AI personalisation vendor, had to outperform product suggestions that people at marko.pl had refined by hand. One early test even pushed overall conversion the wrong way, and the team needed five experiments before the signals began to agree.

In one week long test, the AI widget drew clicks from 16% more sessions than marko.pl’s existing recommendations. Overall conversion came in 5% lower. A result is statistically significant when it is unlikely to be luck, and this gap fell short of that bar. So it cannot show that AI hurt sales. It did expose a trap that many growth teams will recognise. The easiest metric to lift can be the wrong one to chase.

Marko is a Polish family jewellery business that says it has operated for more than two decades, with an online shop and physical stores. According to Witly’s account of the experiment, the retailer used dedicated recommendation modules and manual tagging. Any gain from the algorithm had to come on top of work the store had done.

Clicks pointed in the wrong direction

The first experiment ran for three weeks and compared four product page configurations. Three used different combinations and ordering of Witly’s AI recommendation sections. A fourth removed the AI widget and left marko.pl’s existing store recommendations in place.

One AI configuration generated the most recommendation clicks. The existing recommendation system led on add to cart activity, conversion and revenue. The best clicked AI version also recorded the lowest conversion of the four, though that difference fell short of statistical significance.

Witly says the experiment could reliably detect only a relative conversion difference of about 10%. Marketers call that the minimum detectable effect. Think of it as a ruler with coarse markings. The observed gaps, closer to 3%, were smaller than the smallest mark. Picking a winner after scanning several metrics would have invited selection bias, where the result that looks best partly reflects luck. The team’s working explanation was that a prominent recommendation widget might pull shoppers away from the product they were considering, so they turned that explanation into another test.

Five tests changed the recommendation mix

In the second experiment, the recommendation block moved to a less prominent position. After eight days, purchases attributed to recommendations were about 40% lower and recommendation clicks were down roughly 30%, while overall conversion showed no significant change. Witly stopped that version.

The third test pitted an AI widget with four recommendation sections against marko.pl’s existing recommendations for one week. The AI version lifted the share of sessions clicking recommendations by 16%, yet overall conversion was 5% lower on a relative basis. The movement pointed in a direction without reaching statistical reliability.

Witly then changed the intervention. “Others also bought” moved to the first position, bestsellers were added, and recommendation types suspected of encouraging catalogue browsing were removed. The system also gave more weight to price similarity, which cut the chance that someone looking at a PLN 50 item would be sent toward a PLN 500 alternative.

Witly says its personalisation model learns how an individual differs from the store’s average customer and recommends from that difference. The choice of model affects recommendations. The experiment also suggests that placement, sequence and merchandising logic, meaning the rules about what to show and in which order, can outweigh the model’s contribution.

The final variant produced a coherent signal

In the later configuration, total recommendation clicks were similar across the AI version and the store’s manual system. The AI version drew recommendation clicks from 22.1% more sessions, and Witly reports that purchases following a recommendation were 2.15 times higher, a 115% lift against the existing system.

The manual system appeared to generate repeated browsing from a smaller group. The AI version reached more separate shoppers, each clicking fewer times, and more of those journeys ended in a purchase. The difference provides a better diagnostic than raw click volume, because it asks whether the click helped someone complete a buying decision.

Now the part that separates a learned marketer from a hopeful one. If every extra recommendation purchase had been incremental, meaning a sale that would never have happened without the feature, the implied conversion lift would have been about 7.9%. Overall conversion moved by about 4%, so the vendor estimates that roughly half of the extra recommendation purchases displaced purchases that would have happened anyway. Marketers call that cannibalisation. In plain terms, the feature took credit for sales it only redirected.

The 4% overall conversion increase is the number a growth team owns, and it is also the least certain result. Witly says it did not reach conventional statistical significance. With a baseline conversion rate around 1%, detecting a 4% relative lift would need about one million users in each variant, by Witly’s estimate.

The strongest result still stops short of proof

This is a vendor published case study about the vendor’s product. The source discloses no actual sample sizes, traffic allocation, confidence intervals, order counts, average order values or margins. It also leaves open whether marko.pl analysed the tests independently, whether assignment stayed stable for returning visitors, and whether the final configuration rolled out permanently.

The strongest evidence is narrower than the headline. More sessions engaged with AI recommendations, and recommendation linked purchases more than doubled. The wider 4% conversion movement is a consistent signal that still awaits confirmation as a causal lift. Witly reports that none of the five tests produced a conventionally significant conversion winner.

Revenue was particularly noisy because marko.pl sells products across a wide price range. Witly says winsorising, a method that caps extreme order values so a few big baskets cannot skew the average, removed much of the apparent revenue advantage in some comparisons. For that reason the company argues against using revenue per visitor to decide this test.

Audit the metric before buying the model

The copyable part is the measurement sequence. Compare AI with the recommendation system currently in production, and settle the business metric before launch. Recommendation clicks can show where the funnel changes, while overall conversion or incremental revenue must decide whether the change pays.

Calculate the minimum detectable effect before spending weeks on a test that cannot answer the main question. Measure cannibalisation by comparing recommendation linked purchases with movement across all sales. The comparison shows the practical value of ecommerce A/B testing when the conversion rate moves more slowly than the behaviour around it. Cann’s 19 day holdout ran into a related problem, with results too messy to call.

Smaller stores may never have enough traffic to prove a 3% or 4% conversion lift in one clean experiment. Even so, some proxies mislead more than others. A model can improve the behaviour it directly touches while the business metric stays flat or falls, so build the experiment to catch that gap. Wolt’s recommendation ranker showed the trade in another form, with discovery up and advertising revenue down.

Experiment detail	What the reporting shows
Change	Witly ran five A/B tests, changing widget placement, section order and price similarity, and compared recommendation linked purchases with total conversion.
Reported result	In the final configuration, sessions clicking recommendations rose 22.1% and recommendation purchases rose 115%, with overall conversion up about 4%.
Evidence	Vendor published case study covering the full test sequence. No independent analysis.
Limits	Vendor published. None of the five tests produced a conventionally significant conversion winner. Sample sizes, margins and rollout status were not disclosed.

FAQ

Sources

Witly’s original marko.pl experiment account, published 2 July 2026

Marko company history and store information

Bartosz Jagielski on LinkedIn, author of the Witly case study

Compare the evidence and operating lessons from more AI experiments in our Growth Signal Index, or browse benchmark tooling in the Industry Contents Lab.

Read next

What AI shopping systems still get wrong Our analysis explores why understanding shopper intent remains difficult even when AI improves product discovery.

Compare this with o2’s recommendations for anonymous visitors, where the reported conversion lift also leaves questions about the baseline and control.