Growth Signal Index
The Growth Signal Audit of 51 Business Experiments
11 November 2026 snapshot: Get the CSV · Read the dated methodology. Underlying source-check dates are recorded separately from the snapshot publication date.
Live index. Updated on every new experiment.
Explore what businesses tested, the results they reported and the evidence behind them.
Filter the tests, then open a row for sources and method. “Controlled” means the test used a control group.
| Company | Growth bottleneck | Change tested | Reported result | Growth hack | Details |
|---|---|---|---|---|---|
|
Other
Industry Contents research
|
Test whether a first cold email that leads with a performance statistic or one that leads with the reader's problem earns more attention from a data-trained audience. | Randomise 2,247 researchers between problem-led and statistics-led first emails, holding sender, personalisation, length, validation link and send window constant. | 49.6% vs 38.8% open rate Controlled | In a first cold email, name the reader's problem and keep the proof one click away. | |
|
Reported result in full
Problem-led first emails were opened 49.6% of the time against 38.8% for statistics-led ones, a 10.8 point gap that held in all four campaign pairings, both regions and both products. Clicks were 3.0% against 1.4%.
Evidence limit
Industry Contents research. Opens and clicks only, no sales measured. Subject line and body varied together. Apple MPP and bot opens excluded.
Read the full story ↗
|
|||||
|
Travel and events
Airbnb
|
Guest embeddings could lag recent browsing and searches by almost two days. | Event-triggered inference refreshed the existing Transformer embedding in 10 to 30 seconds. | ~0.3% No control shown | Keep the model fixed and randomise only feature freshness, then judge the stream on a downstream commercial metric and infrastructure guardrails. | |
|
Reported result in full
Approximately one-third of a percent more uncancelled bookings in an online A/B test, as reported by Airbnb.
Evidence limit
Company-reported; sample, allocation, dates, duration, confidence interval, exact p-value, guardrails and cost are not public.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Wine Access
|
Test whether AI could replace or assist three salaried newsletter writers without reducing realised orders and gross profit. | Randomise 23,970 subscribers between human, Claude and Claude-plus-marketer copy while holding the wines and inputs fixed, then apply a net-profit decision rule. | 23,970 subscribers randomly assigned Controlled | Hold the offer fixed, randomise the production workflow, and decide on realised gross profit plus operating cost with brand and accuracy guardrails. | |
|
Reported result in full
Between-arm sales differences were statistically inconclusive. After annualising the two-week estimates and applying labour and software assumptions, the paper projected 8.48% more net profit for Claude and 9.36% more for the hybrid policy.
Evidence limit
Peer-reviewed field study. RCT 3 was registered after it began, had no no-email control, and reports projected annual profit rather than realised savings. Raw data, exact Claude version and full prompt are not public.
Read the full story ↗
|
|||||
|
Media, social and ads
Netflix
|
Personalise new artwork before sparse canvases and newly launched assets accumulate enough interaction history. | A unified model pooled interaction data across five artwork canvases while CLIP embeddings represented what each image contained. | Only the combined model won the online A/B test Controlled | Test the AI feature, its supporting data change and their combination separately before concluding that the feature itself failed. | |
|
Reported result in full
The image-only and pooling-only variants were flat and non-significant. Only the combined system lifted core online metrics, and a later holdback lifted discovery and streaming hours.
Evidence limit
Netflix disclosed test duration and statistical direction but withheld online effect sizes, sample counts, traffic allocation, confidence intervals and implementation cost.
Read the full story ↗
|
|||||
|
Retail and ecommerce
GARCIA
|
Shoppers still could not judge how garments would look after receiving size advice. | AI virtual try-on added above the existing size-me tool in the treatment arm. | 7.06% No control shown | Test visual confidence separately from size advice and decide on contribution margin after returns mature. | |
|
Reported result in full
7.06% higher conversion and 5.54% lower return rate reported by Faslet.
Evidence limit
Company-reported; no visitor denominator, baseline rates, statistical significance or return-maturation window.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Kruidvat, AS Watson Benelux
|
Large baskets pushed the main checkout button below the fold while a two-person CRO team managed more than 100 experiments a year. | Use Contentsquare insight and Kameleoon PBX agents to generate and configure a sticky checkout control, then require human QA before launch. | 30 minutes from insight to launch-ready test Controlled | Automate bounded build and setup tasks, keep human QA, and let a controlled business metric decide rollout. | |
|
Reported result in full
Kameleoon reports 1.3% more checkout starts and 1.0% higher conversion, both at p below 0.01.
Evidence limit
Kameleoon-published case. Sample size, traffic allocation, absolute rates, stopping rule and revenue effect are not published.
Read the full story ↗
|
|||||
|
Retail and ecommerce
eBag
|
Help repeat grocery shoppers complete a long basket without relying on a new visit or discount. | Shopper-level recommendations used live catalogue data and order history, including a cart module for previously bought items missing from the current basket. | 30% of 693,666 orders contained a clicked recommendation No control shown | Use past purchase history in one bounded cart-page module, then reserve a persistent holdout and judge incremental revenue or margin per eligible visitor. | |
|
Reported result in full
Releva reports that 210,119 of 693,666 orders, or 30%, contained at least one product clicked in a recommendation from 1 June to 28 September 2026.
Evidence limit
Supplier-published click attribution with no holdout. The figures do not measure incremental lift, and April and May 2026 were excluded after a counting error.
Read the full story ↗
|
|||||
|
Retail and ecommerce
ASOS
|
Make high-intent search results more relevant without replacing the shopper's query or rebuilding the catalogue. | ASOS brought its recommendation engine into search so recent views, purchases, bag additions and saved items could rerank results for logged-in shoppers. | 140 basis point reported conversion lift No control shown | Test recommendation signals as a bounded reranking layer inside search, keep a persistent holdout and annualise only after publishing the eligible audience and test window. | |
|
Reported result in full
ASOS reports that personalised search lifted conversion by 140 basis points, equivalent to about £31 million in annualised revenue.
Evidence limit
Company-reported result with no published sample, allocation, duration, market, baseline conversion, confidence interval or revenue definition. The £31 million is annualised arithmetic, not disclosed realised revenue.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Smartphonehoesjes.nl
|
Experiment building and analysis constrained the CRO team’s testing capacity across three European storefronts. | Claude Code built most variants, while n8n and Airtable prepared country-level data for an AI proponent, opponent and jury governed by predefined business rules. | 90 experiments in five months No control shown | Automate low-risk building and first-pass analysis, but route conflicting and delayed business metrics to a human reviewer. | |
|
Reported result in full
The company reports about 90 experiments in five months versus 100 to 120 in a prior full year, with analysis taking about five minutes.
Evidence limit
Award entry submitted by the company, with no workflow control group or commercial lift published. The company says the AI jury was often wrong when return rates or extra metrics entered the decision.
Read the full story ↗
|
|||||
|
Telecom and services
Allianz Benelux
|
Allianz needed to reduce avoidable churn among car insurance customers renewing after a price increase. | A randomised 90/10 test compared the standard renewal email with a policy-data-driven personalised video journey built through Lont's governed AI-enabled workflow. | 10.9% relative churn reduction over 100 days Controlled | Bound AI inside approved content and deterministic rules, keep a live holdout on the incumbent journey and judge the system on retention rather than engagement. | |
|
Reported result in full
Lont reports 100-day churn of 11.4% with video against 12.8% in control, a 1.4 percentage point absolute gap and 10.9% relative reduction.
Evidence limit
Lont supplied the platform and published the exact rates. Pebble's implementation account suggests timing, design and a prize draw also changed, and no uncertainty or raw counts are public.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Sweaty Betty
|
Sweaty Betty's syndicated product feed described products in less depth than its website, while manual product-level highlight rules did not scale. | Matched yoga and running product sets compared the normal feed with AI-generated Google Shopping product highlights. | 80% reported yoga revenue gap versus control Controlled | Treat AI enrichment as a product-data intervention, preserve matched item-level controls, and follow the result from impressions through revenue. | |
|
Reported result in full
Athos Commerce reports 80% more yoga revenue and 43% more running revenue for treatment products than their respective controls.
Evidence limit
Result published by Athos Commerce with no public test duration, control size, matching criteria, market, spend, absolute revenue, assignment method or statistical confidence.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Paulmann Licht
|
Paulmann needed to reduce empty search results and manual relevance tuning while increasing revenue from a complex product catalogue. | A six-week 50/50 bake-off compared its incumbent search with Dynamic Yield Experience Search, initially in default configuration. | 26% reported revenue uplift among search users Controlled | Test an AI search system in its default state against the incumbent before tuning, and judge the rollout on revenue with a persistent holdout. | |
|
Reported result in full
Mastercard Dynamic Yield reports a 26% revenue uplift among users exposed to the treatment and a 98% Probability to Be Best.
Evidence limit
Result published by Mastercard Dynamic Yield with no public traffic volume, baseline revenue, absolute revenue, market, assignment unit or conventional confidence interval.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Karaca International
|
Karaca wanted selected German product pages to convert more visitors without changing traffic acquisition. | Split traffic evenly between existing descriptions and Cypien-generated German copy for a Home Enthusiasts segment across 50 products. | 7% relative conversion lift versus control Controlled | Use AI as a cheap variant factory and let a randomized holdout decide whether the content scales. | |
|
Reported result in full
Cypien reports conversion rising from 1.28% to 1.37%, add-to-cart from 3.16% to 3.39%, and bounce rate falling from 23.20% to 17.72%.
Evidence limit
Result published by Cypien with no visitor count, purchase count, confidence interval, test power or revenue per visitor.
Read the full story ↗
|
|||||
|
Retail and ecommerce
BOXIO
|
BOXIO wanted faster Shopify navigation without rebuilding its store and required proof from its own order data. | Split traffic evenly between the original site and Speed Kit's caching, service-worker and predictive preloading layer. | 9.65% higher user conversion versus control Controlled | Require an acceleration layer to win a random traffic holdout on orders before full rollout. | |
|
Reported result in full
Speed Kit reports 9.65% higher user conversion, 10.27% more orders and 43% faster median LCP on accelerated pages.
Evidence limit
Speed Kit-published result. No public duration, sample size, baseline conversion, confidence interval or revenue per visitor.
Read the full story ↗
|
|||||
|
Other
Fischer Future Heat
|
Around 2,000 enquiries a week remained unbooked after seven days of sales follow-up. | Route only day-seven leads to an AI SMS agent that asks three qualification questions and books eligible homeowners into the survey calendar. | 105 qualified home surveys booked in 30 days No control shown | Put automation after a clear human follow-up boundary, check the texts against PECR, and test closed revenue against a random holdout. | |
|
Reported result in full
Setterly reports 19 surveys on day one, 50 in week one and 105 qualified home surveys in the first 30 days.
Evidence limit
Attendance, quote and close rates have not been published. The £630,000 pipeline figure multiplies appointments by average order value.
Read the full story ↗
|
|||||
|
Travel and events
Javo
|
Growth teams need to know whether an AI sales agent adds revenue across all visitors, including the many who never open the chat. | A visitor-randomised design compared no agent, Javo and a frontier-model agent inside the same interface, retrieval layer, tools, prompt and guardrails. | 10% higher revenue per visitor Controlled | Keep the interface, tools, prompt, guardrails and measurement fixed so a growth test changes one part of the AI selling system. | |
|
Reported result in full
Company-affiliated research reports 10.0% higher revenue per visitor against no agent and 21.6% higher engaged conversion against the frontier-model agent.
Evidence limit
Javo's founder wrote the paper. Most sites are anonymous, and the paper reports no independent statistical review, study registration or public replication.
Read the full story ↗
|
|||||
|
Travel and events
Spa Grand Prix
|
Fans picked seats, budgets and extras across many languages, and each choice depended on the last. | Added Jack, a voice and text AI agent that answers questions, shows 360 degree seat views and fills the basket. | 70% higher checkout clickthrough, €539,000 extra revenue anticipated No control shown | Measure paid orders against a holdout before counting anticipated revenue. | |
|
Reported result in full
Napster reports 54,647 conversations in 21 languages over a 62 day sales window, 70% higher checkout clickthrough and an anticipated €539,000 in additional revenue.
Evidence limit
No baseline or comparison group disclosed. The revenue figure is an estimate, not confirmed paid orders.
Read the full story ↗
|
|||||
|
Travel and events
Radisson Hotel Group
|
Travellers shortlist hotels in AI assistants first. Could that discovery lead to direct bookings? | Connected ChatGPT hotel discovery to live property data and a handoff to the Radisson booking site. | Bookings converted at about 1.5 times the organic search rate No control shown | Compare matched audiences before calling a channel gap incremental. | |
|
Reported result in full
OpenAI reports that visitors arriving through the ChatGPT integration converted to bookings at approximately 1.5 times the organic-search rate during July and August 2026.
Evidence limit
Compares two traffic sources with different audiences. Does not show the integration caused the gap.
Read the full story ↗
|
|||||
|
Telecom and services
Atlas24h
|
A chatbot answered product questions but captured no leads and made no sales. | Let eligible visitors buy inside the chat, with structured lead handoff to sales for the rest. | €113,722 attributed revenue from 1,705 contracts in July No control shown | Verify each claimed sale in the operating system, then hold back a slice of traffic to see how many sales would have happened anyway. | |
|
Reported result in full
StartAI reports 1,705 confirmed contracts and €113,722 in attributed revenue from 7,559 conversations in July 2026, with lead-to-contract conversion at 61.4%.
Evidence limit
No control group. Attributed sales do not show additional sales.
Read the full story ↗
|
|||||
|
Software and AI
Algolia
|
A pricing page test needed custom code, and the small development team had other priorities. | Used an AI agent to build and configure two live variants, with people keeping review and launch control. | 15% lift in an unnamed metric No control shown | Use AI to speed up variant building, and keep hypothesis, metric and launch decisions with people. | |
|
Reported result in full
Amplitude reports a 15% lift in Algolia's unnamed target metric and an estimated six-figure pipeline impact.
Evidence limit
Target metric, control, sample and duration are not disclosed.
Read the full story ↗
|
|||||
|
Media, social and ads
Spotify
|
A multi turn recommendation agent needed testing before real multi turn data existed. | Generated replayable synthetic dialogues, audited them with human annotators and used failure traces to improve the agent. | Weekly active users up 5% in a two week test Controlled | Use replayable dialogues and a small human audit to find failures before paying for a live test. | |
|
Reported result in full
Spotify reports 14% more additional listening, 5% more weekly active users, and a 5% lower skip rate in a two-week A/B test reaching about 15 million users.
Evidence limit
Company reported. Markets, baselines and confidence intervals are not published.
Read the full story ↗
|
|||||
|
Media, social and ads
Teads
|
Short, fragmented user histories made sequential targeting hard inside strict bidding time limits. | Fed short event sequences into a reusable user model, tuned by a bounded set of LLM proposed changes. | Click through rate up 2.13% in a seven day test Controlled | Let AI propose changes only from an approved list, and require a live control before claiming growth. | |
|
Reported result in full
Company-reported seven-day A/B results showed click-through rate up 2.13%, visit rate up 2.37%, effective cost per click down 1.13%, and effective cost per visit down 1.47%.
Evidence limit
Company run. Traffic volume undisclosed and 80% confidence intervals used.
Read the full story ↗
|
|||||
|
Software and AI
TreeHouse
|
Manual prompt checks could not link AI search visibility to paid trials across four rental sites. | Linked automated prompt monitoring and crawler logs to referral tracking for paid trials. | AI search drives about 25% of organic conversions No control shown | Tie AI referrals to a commercial event before optimising citations. | |
|
Reported result in full
Supplier Promptwatch says AI Search now accounts for about 25% of organic conversions, with ChatGPT supplying 99% of AI-referred conversions.
Evidence limit
Channel share, not lift. No counts, baseline or control disclosed.
Read the full story ↗
|
|||||
|
Software and AI
Zero
|
Pipeline records went stale because updating them by hand took time, so teams stopped trusting the data. | AI agents read email, calls, usage and billing, then update CRM records. Only outbound messages wait for approval. | Six customer stories, no published error rate No control shown | Time how long it takes to correct the AI written records during a trial before trusting the savings. | |
|
Reported result in full
Six Zero customers report time saved, shorter sales cycles and avoided hires, including at least 50% less pipeline maintenance time at Taito.ai and a sales cycle cut from 60 to 30 days at Watchdog.
Evidence limit
Customer stories only. None explains how savings were measured.
Read the full story ↗
|
|||||
|
Telecom and services
Sterling Insurance Services
|
Shoppers the broker paid to attract hesitated over cover or wording and left before buying. | An AI agent predicted likely exits and answered questions from the live policy wording, starting with two learner driver schemes. | Quote conversion up 31.4% relative over eight months Controlled | Start with one product family whose documents agree, keep a control and watch how the rest of the journey changes. | |
|
Reported result in full
OpenDialog reports a 31.4% relative lift in purchase conversion and a 9.3% lift in add-on attachment over an eight month controlled test.
Evidence limit
Controlled test, but sample size, absolute rates and significance are not disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
RS Group
|
Reject all cookie consent left paid search bidding learning from incomplete customer journeys. | Rebuilt journeys from first party event data and fed the conversion signal into automated bidding in a four month Italy pilot. | 6% more revenue on 5% less spend No control shown | Treat the conversion feed as a growth lever and test it against a stable holdout before scaling. | |
|
Reported result in full
Corvidae reports 6% more paid-search revenue, 5% less ad spend and a 12% ROAS increase.
Evidence limit
Control design, baselines and statistical uncertainty are not disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Galeria
|
Replacing site search with a usage priced AI system meant sales had to outgrow a new variable cost. | Cleaned the product feed, then A/B tested AI commerce search across 1.5 million sessions. | 15% sales lift across 1.5M sessions Controlled | Judge a metered tool on revenue in the same unit as the bill. | |
|
Reported result in full
e-dialog reports a 15% sales increase with stable click rates.
Evidence limit
Traffic split, duration, system cost and profit impact are not disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
The Very Group
|
Fixed triggers showed the same offers to shoppers with very different intent. | An AI agent picked which approved experience each shopper saw, using live intent signals. | 20% more revenue per customer in the first month No control shown | Test the agent against your existing rule and judge it on margin as well as revenue. | |
|
Reported result in full
Very reports a 20% rise in revenue per customer in the first month, plus an unquantified conversion gain.
Evidence limit
Baseline, control design, margin and discount cost are not disclosed.
Read the full story ↗
|
|||||
|
Telecom and services
Telefónica Deutschland (o2)
|
The most valuable visitors were anonymous, so profile based rules could not follow a changing session. | Joined live session behaviour with product feed data and AI recommendations to refresh home page hero banners. | Conversion up 275% relative on personalised banners No control shown | Validate one narrow placement first and keep a holdout on the old rule. | |
|
Reported result in full
e-dialog reports a 275% conversion rate uplift and a 17% click through rate on personalised hero banners.
Evidence limit
Relative figure only. Baseline conversion, control and test dates are not disclosed.
Read the full story ↗
|
|||||
|
Travel and events
Robinson Club (TUI Group)
|
Landing pages and competitor research took one or two days, slowing the start of each test. | Taught an AI the brand voice, page frameworks and CMS components, then added agents to check its output. | Page build time cut from days to seconds No control shown | Give AI only the page parts and checks you already trust, then time each test from request to launch. | |
|
Reported result in full
Optimizely reports page work falling from days to seconds.
Evidence limit
No conversion result for an AI built page and no manual control disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
marko.pl
|
AI recommendations had to beat a tuned manual setup, and early click gains did not match conversion. | Ran five A/B tests on widget placement, section order and price similarity. | Recommendation purchases up 115%, overall conversion up about 4% Controlled | Set the deciding metric before launch and check whether recommendations took sales from elsewhere. More clicks can hide flat sales. | |
|
Reported result in full
In the final configuration, sessions clicking recommendations rose 22.1% and recommendation purchases rose 115%, with overall conversion up about 4%.
Evidence limit
No test produced a conventionally significant conversion winner.
Read the full story ↗
|
|||||
|
Software and AI
Fyxer
|
Building an experiment took days or weeks, limiting how many tests ran. | Built a reusable AI workflow linking shared instructions, coding agents, internal data and the experiment API. | 360 tests by four engineers in a year No control shown | Find the slowest repeatable step between idea and decision and hand AI one narrow job there. | |
|
Reported result in full
Four growth engineers ran 360 tests in one year.
Evidence limit
Company wide test totals conflict with the practitioner account. No sample sizes disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Zalando
|
Test and control groups competed for the same advertiser budgets. | Split each campaign budget into virtual sub campaigns before the auction, so simultaneous tests can share campaigns. | Experiments up from 8 to 60 plus a year No control shown | List what test and control share, such as budget, and split it before the test starts. | |
|
Reported result in full
Zalando reports experiment volume rising from eight in 2023 to more than 60 in 2025.
Evidence limit
Staffing, success rate and causal credit for the increase are not disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Deliveroo
|
A high recall model raised more churn alerts than the retention team could act on. | Ranked restaurants by churn risk, set a threshold to match team capacity and planned a randomised holdout. | Alerts down over 90% offline, precision up eightfold No control shown | Send only as many alerts as the team can act on, and keep a comparable group untouched to learn whether outreach keeps customers. | |
|
Reported result in full
Offline alerts fell by more than 90% and precision improved eightfold. The churn experiment is still running.
Evidence limit
Offline result only. The live churn effect is still being tested.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Matas Group
|
The loyalty team could not tell which dormant subscribers to target, or prove that re engagement changed retention. | Used explainable AI to find a dormant cohort and withheld the re engagement flow from a control group. | Retention up 29 percentage points after 30 days Controlled | Keep audience selection separate from treatment assignment and preserve a control to measure incremental retention. | |
|
Reported result in full
Subsets reports a 29 percentage point retention lift after 30 days, plus 39% higher web engagement and 20% more orders.
Evidence limit
Sample size, baseline rates and confidence interval are not disclosed.
Read the full story ↗
|
|||||
|
Telecom and services
Orange France
|
A universal €10 offer leaked discounts to shoppers who would have bought anyway. | A model showed the offer only to hesitant visitors, tested against no offer and the universal offer. | Promo code use down from 68% to 25% of purchases Controlled | Test a targeted discount against both no offer and a blanket offer, and judge it on margin, not conversion alone. | |
|
Reported result in full
Kameleoon reports conversion up 11.6% against no code and 7.6% against the universal offer, with promo-code use falling from 68% to 25% of purchases.
Evidence limit
No independent analysis or profit data.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Wolt (DoorDash)
|
Purchase history favours reorders and crowds out new merchants. | Added an adjustable weight to a venue ranker that favours first time merchant orders. | Merchant trial rate up 5.5%, ad revenue down Controlled | Make discovery an explicit setting, then check what it costs reorders and neighbouring revenue such as advertising. | |
|
Reported result in full
Merchant trial rate rose 5.5% and global conversion 0.16% in the first test. Advertising revenue declined, by 0.43% under the final version.
Evidence limit
Company paper. Sample counts, absolute rates and the value of the order gain are not disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Allegro
|
Co purchase data confused complementary products with alternatives, and results varied by placement. | Tested category guided recommendations on product page, pre cart and cart placements. | Cart GMV up 21.25% on web, up 15.73% in app Controlled | Test recommendations in each placement, because a model that lifts cart sales can fail one step earlier. | |
|
Reported result in full
+21.25% attributed cart GMV on Web; +15.73% in the app
Evidence limit
Counts sales attributed to carousel clicks, not platform wide revenue. Pre cart GMV stayed flat.
Read the full story ↗
|
|||||
|
Software and AI
SignalTo.ai
|
A new ad platform had no public benchmarks and two offers to compare. | Ran a question led ad and an offer led ad in ChatGPT, with context hints for targeting. | 7 report requests, all from one ad No control shown | Creative set the price of a click, but seven leads only show a direction. | |
|
Reported result in full
7 report requests, all from one question led ad
Evidence limit
Founder reported. The free report ad had the most impressions and zero leads, and 27 clicks is a small base.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Proper Cloth
|
Click based analytics and Google view through reports disagreed on whether YouTube drove extra sales. | Ran a five week geo holdout, with YouTube Demand Gen live in three states against a synthetic control. | About $180,000 estimated sales lift (12.7%) Controlled | When attribution systems disagree, design a counterfactual test and use the result to pick the next experiment, not a permanent verdict on the channel. | |
|
Reported result in full
Estimated 12.7% incremental sales lift, about $180,000 (95% credible interval roughly $98,000 to $255,000), with a 2.43 return on ad spend.
Evidence limit
Model estimate. Spend, state selection and pre period fit are not disclosed.
Read the full story ↗
|
|||||
|
Media, social and ads
Douyin (ByteDance)
|
Incentive offers had to be priced before the platform knew if users would finish ads and cover the payout. | Used offline reinforcement learning with a world model to rank incentive policies before four online tests. | Net profit per user up 7.96% Controlled | Judge incentives on net profit at matched cost and treat ad exposure as a diagnostic. | |
|
Reported result in full
The world-model policy lifted per-user revenue 6.80% and net profit 7.96%. Removing it raised ad exposures 23.68% but cut revenue 4.77% and net profit 6.56%.
Evidence limit
Company preprint. Absolute revenue, profit, incentive values and dates are not disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Zalando
|
Sales events lasted about six days but the pricing forecast ran weekly, and decisions took hours of manual work. | Daily demand forecasts feeding an optimiser that balances revenue and long term profit. | Profit contribution up 6% across 12 markets Controlled | Choose a forecaster by the decisions it produces after optimisation, not its standalone error score. | |
|
Reported result in full
+6% profit contribution (95% CrI 0.79% to 11.03%), revenue and sales statistically unchanged
Evidence limit
Company preprint with no absolute figures. Revenue and sales were statistically unchanged.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Taobao Live (Alibaba)
|
Compact AI hosts memorised skill names and prompt layouts, so every rule change risked breaking them. | Trained a model on shifting skills, tools and prompts, with an editable rule set that operators update without retraining. | GMV per participant up 4.33% across 1.98M users Controlled | Move the sales playbook out of the model weights so operators change rules in hours, then test the whole system. | |
|
Reported result in full
+4.33% confirmed-receipt GMV per participant, +0.91% item page views
Evidence limit
Dashboard snapshot shows one significant day per metric and no confidence intervals.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Kuaishou
|
Designing recommendation experiments by hand was slow, and a win on one metric could quietly hurt the platform. | An AI agent proposed new strategies, drawing on 310 past ones, over five rounds of live tests. | GMV up 4.8% by the fifth strategy Controlled | Give an experiment agent hard guardrails, because a win on the target metric can cost the rest of the platform. | |
|
Reported result in full
+4.8% GMV after five rounds, with platform guardrails held
Evidence limit
The first two strategies lifted GMV but broke platform guardrails. Company paper.
Read the full story ↗
|
|||||
|
Software and AI
Wantedly
|
Candidates with few recent recruiter approaches churned more, yet ranking kept sending exposure to popular candidates. | Boosted ranking for candidates with fewer than three recent scouts if they still cleared a match score threshold. | Churn odds ratio 0.957, not statistically significant Controlled | A directional result that misses significance still shows where the evidence chain breaks. | |
|
Reported result in full
Churn odds ratio 0.957 (p=0.128), directionally lower, not statistically significant
Evidence limit
Not significant (p=0.128). Total matches up 5.4%, also not significant.
Read the full story ↗
|
|||||
|
Other
AMap (Alibaba)
|
The homepage already had a strong ranker for the next stop, which was hard to beat without slowing the app. | Replaced the ranking stack with a small AI model that predicts the next stop directly. | Active user rate up 9.67% over one month Controlled | The most accurate offline model is not always the one to ship. The smallest version that fits the speed budget can be the right call. | |
|
Reported result in full
+5.83% page CTR (1wk), +9.67% active-user rate (1mo)
Evidence limit
No confidence intervals or baselines published.
Read the full story ↗
|
|||||
|
Media, social and ads
Kwai
|
Ads were marked as underperforming just for converting a little late. | Retrained the scoring system to judge ads by whether a sale eventually happens, including late conversions. | Realised ad revenue up 1.86% Controlled | Split conversion prediction into two clocks so a late sale is not scored as a failed click. | |
|
Reported result in full
+2.49% expected revenue, +1.86% realized revenue, +2.06% conversions
Evidence limit
Company preprint, not independently verified.
Read the full story ↗
|
|||||
|
Media, social and ads
The Guardian
|
Most app users skipped sign in without a second thought. | Made the skip option harder to spot on the first screen. | Registrations and sign ins up 80%, agency reported Controlled | Hiding the skip option lifted sign ins, but the case cannot say if those accounts stuck. | |
|
Reported result in full
+80% registrations & sign-ins
Evidence limit
Agency case study. No baseline or sample size, and no data on whether new accounts stayed.
Read the full story ↗
|
|||||
|
Media, social and ads
Meta
|
Attribution credited ads for sales they had no real hand in, such as buying a different item. | Changed how a sale is counted when someone clicks one item but buys another. | Same item purchases up 1.27% in a two week test Controlled | A conversion can be real while its attribution label is wrong. Separate same item and cross item purchases before optimising on them. | |
|
Reported result in full
Online, +0.25% in a proprietary primary business metric and +1.27% same-item purchase rate. Offline, normalized entropy was 13.9% lower than last-click.
Evidence limit
The 13.9% offline figure is a prediction measure, not a lift. Audience and baselines undisclosed.
Read the full story ↗
|
|||||
|
Media, social and ads
ShareChat
|
Revenue experiments needed lots of traffic and time before results could be trusted. | Grouped users by spending history before testing, to cut noise from heavy spenders. | About 45% less traffic needed per revenue test Controlled | When a few heavy spenders drive revenue variance, split users by spend history before testing instead of adding traffic. | |
|
Reported result in full
~45% less traffic needed, ~50% faster decisions
Evidence limit
Company paper validating the method on 40 plus tests, not independently replicated.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Bunzl
|
The most clicked homepage section was quietly steering shoppers the wrong way. | Replaced that section with customer reviews. | Sales up 9%, product page visits up 11% Controlled | High clicks on a section do not prove it helps. Check what visitors do next before defending it. | |
|
Reported result in full
+11% product-page visits, +9% sales
Evidence limit
Provider case study. No baseline or significance disclosed.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Sport Chek
|
Would a winning shipping message survive a higher spending requirement? | Raised the free shipping threshold from $50 to $75 with the same message. | Up 6.56% at $50, down 15% at $75 Controlled | Retest the message when the offer changes. The previous winner no longer describes the same purchase decision. | |
|
Reported result in full
+6.56% at $50 threshold, -15% once threshold rose to $75
Evidence limit
The offer changed alongside the message, so the two effects cannot be separated. No sample size disclosed.
Read the full story ↗
|
|||||
|
Media, social and ads
LinkedIn
|
Ads reached people likely to notice them, with little regard for who the ads could actually sway. | Built a system that targets people the message has a real chance of changing. | Long term value up 7.2% over eight weeks Controlled | A response does not prove the message caused it. Target for incremental effect and withhold sends predicted to add nothing. | |
|
Reported result in full
+7.20% long-term value (95% CI, 0.31% to 14.09%)
Evidence limit
Company preprint. Absolute baseline undisclosed, and the 95% interval runs from 0.31% to 14.09%.
Read the full story ↗
|
|||||
|
Software and AI
Aurum (Astrea)
|
New trial users waited about two days before their data showed up. | Cut the wait so real data appears the moment a trial starts. | Trial engagement up 125% relative, 16 points Controlled | Preloading data into the first session lifted activation, but engagement means little without downstream conversion. | |
|
Reported result in full
+125% relative (16pp) clipping engagement
Evidence limit
Provider case study. No baseline, sample size or paid conversion data.
Read the full story ↗
|
|||||
|
Media, social and ads
Pinterest
|
Countless photos had no clear reason to appear in search. | Gave every photo an AI written description built for search. | Visits up 18% in a four week test Controlled | Writing the search intent for an image, not just its label, carried most of the extra traffic. | |
|
Reported result in full
+18% visits (four-week test)
Evidence limit
Company paper. Traffic allocation and sample counts are not disclosed.
Read the full story ↗
|
|||||
|
Software and AI
Outrank
|
Could a paid leaderboard spot bring in sign ups that justify the cost? | Spent about $12,000 on the placement and tracked trial sign ups. | 53 extra trials on about $12,000 No control shown | A before and after spike is a correlation, not a controlled test. The real result depends on how many trials become paying, retained customers. | |
|
Reported result in full
53 trials above baseline on ~$12k spend, paid conversion unproven
Evidence limit
No control group. Trials rose but paid conversion is unproven.
Read the full story ↗
|
|||||
|
Retail and ecommerce
Cann
|
A top ad channel was suspected of looking better on paper than in practice. | Turned the channel off across five states for 19 days to see what changed. | About $90k estimated lift, range from a $152k loss to a $310k gain Controlled | An inconclusive incrementality test still gives a better basis for a budget decision than a dashboard ROAS figure. | |
|
Reported result in full
~$90k est. lift, 95% CI spans -$152k to +$310k
Evidence limit
The range is too wide to say whether the channel drove sales. Third party case study.
Read the full story ↗
|
|||||
| No tests match those filters. | |||||
Swipe or scroll sideways to see the full table on mobile.
How to read this index
Each row is one Growth Signal story: the bottleneck, the change tested, the reported result and the limit on the evidence. Results are as reported by the company or agency involved and are not independently verified unless the story says so. Controlled means the source describes a control group or a randomised comparison. No control shown means the figure is a company report, a before and after change or an attribution count, which does not establish extra sales. Open a row for the test design, the source and the learning to carry into your next test.
Cite this data
Industry Contents, Growth Signal Index, IC Lab, https://industrycontents.com/ic-lab/growth-signal-index/, accessed [date].
Growth results can change as more evidence appears. Check the source story before quoting figures elsewhere.
Get the Growth hack digest
One growth experiment, its result and the learning to copy, in your inbox. The evidence limits are always stated.
How to read a growth experiment
What counts as a growth experiment?
A change to a product, page, message or channel, tested against a stated goal with a measurable outcome. A good one starts with a hypothesis and ends with a learning that decides the next test.
What is the difference between a reported gain and incremental growth?
A reported gain is a number a company published, such as attributed revenue or a higher click rate. Incremental growth is what would not have happened without the change, and only a control group or holdout can show it. The index marks which results had one.
Can I reuse these A/B test examples?
Reuse the idea, not the number. Your audience, traffic and offer differ, so treat each learning as a hypothesis to test on your own funnel.
Citing and reusing the index
Cite this index with credit and a link to Industry Contents. To copy, republish or redistribute the dataset, request written permission.