Paulmann tested AI search against its own for six weeks and reported 26% more revenue

A shopper hunting for one part of a lighting system can be precise and still baffle a search box. The part might belong to a rail family, carry a technical number and sit under a term the catalogue spells differently. One missed synonym or typing slip can turn a person with an object in mind into a visitor staring at an empty results page.

At Paulmann Licht, preventing those dead ends had grown into a manual job. The ecommerce team kept adjusting synonyms, field weights and business rules as the catalogue expanded. With more than 3,000 lighting products and systems, every fix solved one known problem while the next query waited its turn.

So the German manufacturer set up a race. It put its existing search beside an AI semantic search system, divided traffic evenly and let revenue pick the winner. After six weeks, Mastercard Dynamic Yield, which supplied the challenger, reported 26% higher revenue among users exposed to its search system. The question was small enough for a growth team to own. Could a default model understand enough customer intent to beat years of manual tuning before Paulmann had customised it at all?

When relevance became a maintenance queue

Paulmann Licht is a family owned lighting manufacturer based in Springe, Germany. It employs about 400 people across Europe and sells through several country shops. That breadth helps a customer planning a room. It is harder on a keyword engine, which has to tell a whole system from one compatible piece.

According to the September 2026 case study published by Mastercard Dynamic Yield, Paulmann’s incumbent search needed regular changes to synonyms, weighting and commercial rules. Empty result pages created dead ends, and product relationships made things harder because many items belong to a wider lighting system.

For anyone who thinks in funnels, notice where the problem sat. It came after acquisition. Paulmann had paid for, earned or kept the visit, and search then had to turn an imperfect query into a product worth considering. When search fails, extra traffic only sends more people to the same blank shelf.

The hypothesis was that AI semantic search could infer intent without waiting for an exact catalogue phrase. Better interpretation should cut empty searches, improve product discovery, lift revenue and reduce manual maintenance. That last piece matters. A search tool that converts better but needs a bigger tuning team improves one line of the growth model while worsening another.

Half the search traffic kept the old engine

Paulmann and Mastercard Dynamic Yield began with a workshop to agree business objectives, metrics and the test method. They then ran a six week ecommerce search A/B test. Half of eligible traffic got Paulmann’s existing search, and the other half got Dynamic Yield Experience Search in its default configuration.

The treatment came as a bundle. It covered AI semantic search, query understanding, auto suggest, typo tolerance, recognition of stock keeping units and product names, and merchandising features such as redirects and category suggestions. The algorithm tried to match the likely meaning of a query to products even when the words differed from the catalogue.

Starting with the default configuration is the part I would steal. It gave the teams a clean operational question before anyone spent time improving individual queries. A company needs no internal machine learning group to copy the structure. It needs two search experiences, persistent traffic allocation and commerce events that join back to exposure.

The public account says the default system established a performance advantage, and only after that baseline did the teams refine a small number of high volume queries. It leaves open when the six week measurement window ended relative to those refinements. The headline result may describe a purely out of the box model, or a test period that later included limited human tuning.

Either way, the order of events was sensible. Measure the underlying technology first, then add operator judgement. A team that tunes the treatment heavily before the first comparison cannot tell whether it bought a better model or funded a concentrated editorial clean up.

The revenue result belonged to search users

Mastercard Dynamic Yield reports a 26% search revenue uplift among users exposed to Experience Search. It also reports a 98% Probability to Be Best, the platform’s Bayesian measure of the chance that the treatment outperformed the control under its model.

Look at the denominator in that first figure.

The published result covers users served the search experience. It is presented as a lift for those users and the source does not say how many visitors used search. If one shopper in ten searches, a large effect inside that group can shrink into a much smaller site wide change.

Still, the metric is the right kind. Revenue sits beneath no result rate, search clickthrough and product views. Those mechanism metrics help explain why a treatment worked, and revenue shows whether better retrieval created commercial value or simply more interaction.

Read the 98% figure with care too. Probability to Be Best is a different statement from a frequentist 95% confidence interval, though both try to express uncertainty. The case publishes no Bayesian prior, loss threshold, posterior distribution or absolute revenue values. It calls the outcome statistically significant, and an outside reader cannot reconstruct that judgement.

Paulmann found the evidence strong enough to continue. The company is exploring query optimisation in more countries, Affinity Boost, search within product listing pages and visual search. Robin Petersen, Paulmann’s head of ecommerce D2C, says the result stood out because the system produced the lift with little configuration.

One treatment hid several possible winners

The reported 26% cannot tell you that semantic understanding alone caused the gain. Auto suggest could have shortened queries. Typo tolerance could have rescued misspellings. SKU recognition could have helped high intent trade customers, and merchandising rules could have pushed useful categories. The experiment proves the value of the package against the incumbent and says nothing about each part.

The case also leaves out sample size, baseline revenue, conversion rate, average order value and the share of queries that returned nothing. The first market is unnamed, and there is no device split, query class analysis or word on whether returning users stayed in the same variant. Mastercard owns Dynamic Yield, supplied the treatment and published the case. Paulmann’s named practitioner adds credibility, while the raw results and analysis stay private.

European conditions add a further limit. Paulmann serves country specific shops and says it wants to extend the system across languages and local assortments. A German relevance model will not automatically carry over to a French or Polish catalogue where product vocabulary, synonyms and availability differ. Paulmann’s privacy notice also makes non essential cookie use depend on consent, so anyone reproducing the test should record how consent status affects assignment and measurement.

None of these gaps makes the test useless. They shape the next decision. Paulmann learned that the managed search bundle could beat its incumbent under live conditions, and another team should borrow the bake off and leave the 26% out of its forecast.

The next test should measure the whole shop

A stronger follow up would keep user level assignment for the full test and publish a simple funnel for each group. The stages are search used, results returned, product page opened, item added to cart and purchase completed. Query classes such as product number, product family, room, attribute and misspelling would show where AI semantic search earns its keep.

Revenue per search user stays the main mechanism metric. Revenue per site visitor sits beside it as the business metric, because it weights the effect by the share of people who use search. Conversion, average order value, no result rate and gross margin would explain the path, with returns, page latency, consent loss and irrelevant result complaints as guardrails.

A component test could then compare three groups. The first keeps the incumbent. The second adds typo tolerance, auto suggest and SKU recognition without semantic ranking. The third adds the full Dynamic Yield Experience Search treatment. That design would show whether the model understands intent better, or whether most of the value comes from fixing ordinary search hygiene.

The wider lesson is less glamorous than installing an intelligent search box. Paulmann ran a fair race before it customised the challenger, judged the result on paid behaviour and kept the old engine alive long enough to give a counterfactual. AI made the variant possible, and the holdout made the claim worth reading.

If you have a catalogue and a queue of synonym fixes, put the managed system behind half the traffic, resist tuning it until the first baseline is recorded, and let revenue settle the argument. A search engine can be very good at finishing a customer’s sentence, and it should still have to finish the test.

Experiment detail	What the reporting shows
Constraint	Manual synonym, weighting and business rule changes struggled to keep pace with a complex lighting catalogue.
Intervention	A 50/50 test compared Paulmann’s incumbent search with Dynamic Yield Experience Search, initially in default configuration.
Duration	Six weeks in one undisclosed initial market.
Reported result	Mastercard Dynamic Yield reports 26% higher revenue among users exposed to the treatment and a 98% Probability to Be Best.
Limits	No traffic volume, baseline, absolute revenue, conversion, assignment mechanics, market or conventional confidence interval is public.

Sources and disclosure

• Mastercard Dynamic Yield’s September 2026 Paulmann Licht case study
• Mastercard’s web version of the case
• Paulmann’s company profile and data protection declaration
• Paulmann head of ecommerce D2C Robin Petersen on LinkedIn

Mastercard owns Dynamic Yield, supplied the search system and published the result. Paulmann’s ecommerce leader is quoted by name, and the retailer has not released the underlying dataset, the analysis or an independent account of the experiment.

FAQ

Compare more measured experiments in the Signal Index, browse the Growth Signal archive, or use the benchmark tools in the IC Lab.