7 min read
TL;DR
- In Zalando’s retail media business, the test group and the control group drew from the same campaign budgets, so a winning variant could drain the budget the other variant needed.
- Zalando calls this budget cannibalization bias, a form of interference that breaks the usual A/B test assumption.
- Its Budget Split tool divides each campaign into virtual sub campaigns before the auction, in proportion to the traffic split.
- Orthogonal concurrency lets simultaneous tests share campaigns, and Zalando reports experiments rising from eight in 2023 to more than 60 in 2025.
- Shared model training history can still contaminate results, and Zalando leaves out success rates, durations and how much of the increase Budget Split alone explains.
Zalando says splitting budgets lifted its yearly ad tests from eight to over 60.
Picture two children sharing one jar of pocket money. The quicker one spends most of it, and the slower one ends up looking like the worse saver. An advertising experiment can fall into the same trap, and the dashboard will still look clean.
That happened inside Zalando Partner Marketing Services (ZMS), the retail media business of the European fashion platform. Retail media means advertising sold on a retailer’s site, and advertisers there compete for placements with finite campaign budgets. When a new bidding feature and the old system drew from the same wallet, the better performer could spend the money first. The weaker version then had less budget left to compete.
So the apparent treatment effect reflected more than the feature itself. Part of it came from one test arm starving the other.
A shared wallet broke the control group
Zalando describes the problem in a September 2026 engineering account by senior product manager Stefano Restelli and senior applied scientist Carlos García Alvarado.
Start with how an A/B test works. The treatment group gets the new feature and the control group keeps the old one. The comparison only holds if one group cannot change the other group’s outcome. Retail media breaks that rule, because advertisers, shoppers and auction systems share limited money and inventory. A treatment that wins more auctions burns through budget faster. Control users may then see fewer eligible bids, even though the control product never changed.
Zalando calls this budget cannibalization bias. It is a form of interference between test groups, and it goes well beyond cosmetic reporting. When the groups affect each other, the experiment cannot cleanly isolate the causal effect of the feature. ShareChat met a different measurement trap when a few big spenders distorted its A/B tests.
Test velocity was the growth constraint. Zalando says experiments had become flaky and had to run one after another, because simultaneous tests competed for traffic and campaign budgets. The company recorded eight experiments in 2023 without its new isolation capability.
Zalando split budgets before the auction
The company built Budget Split, which divides every participating campaign into virtual sub campaigns before bidding begins. Users are randomised into treatment or control, and the campaign budget is partitioned in the same proportion as the traffic. In the pocket money picture, each child now gets a separate jar.
Treatment users trigger bids only from the treatment budget. Control users trigger bids only from the control budget. Each variant therefore operates inside its small marketplace instead of fighting over one wallet.
That upstream split is the practical lever. Growth teams running two sided marketplace experiments can list every scarce resource shared by treatment and control before launching a test. Money is one example. Inventory, delivery capacity, lead supply and sales attention can create the same interference.
Separating the scarce resource is harder than randomising the user. Even so, user randomisation alone cannot fix a marketplace test when the underlying supply stays shared.
Concurrent tests multiplied the budget pools
One isolated A/B test still left Zalando’s concurrency problem unsolved. Two simultaneous experiments create four possible treatment combinations. The team introduced orthogonal concurrency, which splits campaign budgets into matching buckets such as 1A2A, 1A2B, 1B2A and 1B2B. Read each label as the version a shopper sees in test one, then in test two.
The architecture preserves the combinations needed to estimate each feature’s effect while letting tests run at the same time. Zalando says ZMS went from eight experiments in 2023 to more than 60 in 2025 after adding the isolation properties described in its post.
That is a reported rise in experiment volume. It stops short of showing that every decision became correct or that revenue grew by the same proportion. Zalando publishes no test durations, staffing changes, experiment success rates or the share of the increase that Budget Split alone explains. Read the result as evidence that the infrastructure removed a constraint on testing.
The machine learning models could still leak history
Budget isolation left some contamination in play. Zalando’s predicted click through rate model retrains daily on seven days of history, while its predicted conversion rate model uses 28 days. These models guess how likely an ad is to earn a click or a purchase, and they learn from past behaviour. Pre experiment events can therefore carry the old system into the treatment model.
This creates machine learning contamination because separate live budgets still share training data. A treatment model may learn from behaviour the control era system produced. Zalando says its architecture can support separate training pipelines, and the company keeps monitoring pre experiment data distributions.
Other interference remains. Automated campaign steering can adjust budgets or maximum bids using combined performance. Advertising staff can react by hand to aggregate results. Campaigns that outlast the experiment window can also skew revenue interpretation.
When Zalando cannot guarantee isolation, the team says it turns to causal inference methods instead of forcing an A/B test. One example is non experimental counterfactual analysis, which estimates what would have happened without the change.
The audit starts with shared constraints
Nobody needs Zalando’s auction stack to borrow the lesson. Draw the interference path before accepting a test design, and run these five checks.
- Name the finite resource that treatment and control share.
- Check whether one variant can consume or reshape that resource.
- Partition it upstream when the business can support separate pools.
- Inspect model training windows for data produced before randomisation.
- Use a counterfactual method when isolation cannot be defended.
This also changes the metric a growth operator should track. Test count helps, yet a better pair is decision speed and the rate of experiments that pass interference checks. More tests create value only when their control groups stay credible.
Zalando’s case shows that experimentation infrastructure belongs to the growth system. The expensive mistake sat deeper than a slow dashboard. The product under test was allowed to alter the conditions under which its alternative was measured.
FAQ
How does budget cannibalization bias happen?
It is interference between test groups. A treatment that wins more auctions burns through the shared campaign budget faster, so control users see fewer eligible bids even though the control product never changed.
How does Budget Split work?
It divides every participating campaign into virtual sub campaigns before bidding and partitions the budget in the same proportion as the traffic. Treatment users bid only from the treatment budget and control users only from the control budget.
How did running multiple tests at once work?
Orthogonal concurrency splits campaign budgets into matching buckets such as 1A2A, 1A2B, 1B2A and 1B2B, which preserves the combinations needed to estimate each feature’s effect while letting tests overlap.
Does Zalando say how much Budget Split explains?
It does not. The account leaves out test durations, staffing changes, experiment success rates and how much of the rise from eight to more than 60 experiments Budget Split alone explains. It also notes remaining contamination from shared model training history.
Sources
- Zalando Engineering on Budget Split and orthogonal concurrency
- Zalando Partner Marketing Services
- Stefano Restelli on LinkedIn
- Carlos García Alvarado on LinkedIn
Compare the evidence and operating lessons from more AI experiments in our Growth Signal Index, or browse benchmark tooling in the Industry Contents Lab.
Read next
What AI shopping systems still get wrong Our analysis explores why understanding shopper intent remains difficult even when AI improves product discovery.
For another threat to experiment reliability, read how a few big spenders distorted ShareChat’s A/B tests.