Experiment summary
- Kuaishou built an A/B Agent that studied 310 past recommendation strategies, proposed configurations and learned from each live result, with a human reviewing before launch.
- Strategy 1 lifted GMV 1.123% and Strategy 2 lifted it 2.984%, but both moved the reported guardrail metrics in the wrong direction.
- After the agent changed objective weights and thresholds, Strategy 5 became the operating point with every disclosed measure positive.
- The company reports GMV up 4.829% against production, and 2.150 percentage points better than an expert designed strategy.
- Traffic split, duration, baselines and confidence intervals were not disclosed, and the authors work for Kuaishou.
Kuaishou’s recommendation engineers had a familiar growth problem. Improving the shopping feed required repeated experiments, but every round still depended on an expert to retrieve old tests, design the next strategy, read the results and decide what to change.
Kuaishou tested whether an AI agent could take over more of that loop. It did not ask the system to write ad copy or recommend products directly. It asked the agent to study 310 previous recommendation strategies, propose a production configuration, learn from each A/B result and choose the next move.
In a Kuaishou-authored paper published on 5 August 2026, the team documents five successive strategies on its short-video ecommerce platform. The final configuration produced the strongest reported business result, with gross merchandise value up 4.829% against the production comparison. The experiment is more useful for what happened before that number, the first two strategies raised the target metric while damaging wider platform measures.
Recommendation work was stuck in a manual loop
Kuaishou operates short-video, livestreaming and ecommerce services. Its shopping recommender has to decide which commercial videos users see while protecting the broader experience that keeps them watching, browsing and buying.
A recommendation strategy can span retrieval, ranking, blending, scoring rules and parameter settings. Changing one part may improve ecommerce GMV while reducing watch time or revenue elsewhere on the platform. The paper says engineers traditionally spend hours analysing a typical experiment and selecting the next configuration.
The constraint was therefore not a shortage of tests. It was the time and specialist judgement needed to connect them. Historical knowledge sat across documents, code, configurations and metric logs. Similar experiments could still differ in user intent, available features, pipeline stages and guardrails.
The documented idea behind the Kuaishou A/B Agent was to turn those scattered records into reusable operating knowledge. The system would retrieve relevant precedents, generate an executable strategy and then use live experiment results to revise its next move.
The agent organised 310 old strategies
The team assembled an industrial benchmark from 310 historical strategies across three Kuaishou ecommerce scenarios. These covered retrieval, ranking, blending and other recommendation stages.
Kuaishou is framing this as automation of the experimentation process itself, a claim to compare with how other vendors sell automation. Our deeper analysis of UiPath and a 15-person rival selling two different ideas of automation lays out how those vendor claims tend to diverge from what ships.
Instead of storing each experiment as an isolated document, the agent broke it into a motivation, mechanism, parameters, applicable context, outcome and risk. It then arranged those pieces in a hierarchical tree by business domain, scenario, recommendation stage and objective.
When the team supplied a new optimisation request, the system identified the scenario, target metric, guardrails and deployment constraints. It retrieved several possible precedents and used GLM-5.1 to generate a proposed configuration. A human engineer reviewed the strategy before it entered the online testing platform.
There was an approval step. The AI recommendation strategy was not deployed without oversight. The paper describes an approval step before launch, although it does not say how often engineers rejected or modified the agent’s proposals.
Once a test ran, the agent collected the core and guardrail metrics, compared the new version with its parent and sibling experiments, and recommended another parameter change. Promising branches triggered local tuning. A saturated branch or a guardrail failure could send the system back to retrieve a different mechanism.
The first gains came with hidden costs
The five-round sequence is the clearest evidence in the report. Strategy 1 increased GMV by 1.123%, but its four reported guardrail metrics declined by 0.405% on average. Watch time, live GMV and platform GMV all moved in the wrong direction.
Strategy 2 pushed GMV to 2.984% and produced the largest click gain in the sequence, at 1.167%. Every guardrail remained negative. A team looking only at ecommerce conversion optimisation could have mistaken the larger target lift for a clean win.
The agent then changed the objective weights and decision thresholds. Strategy 3 raised GMV by 3.299% while the average guardrail change became positive for the first time, at 0.223%. Live GMV still declined slightly by 0.067%.
Strategy 4 held GMV near the same level at 3.253% and kept the guardrail average positive. The authors describe this round as a stability check rather than a meaningful regression. Strategy 5 became the selected operating point, with every disclosed measure positive.
This progression makes the test more informative than a single result. The early variants show why automated A/B testing still needs business guardrails. The system found configurations that improved the local target before it found one that avoided visible damage elsewhere.
The final strategy raised ecommerce GMV
Kuaishou deployed the final strategy in its ecommerce short-video recommendation system. Against the production comparison, the company reported GMV up 4.829%, gross profit measure up 4.677%, order profit measure up 0.841%, conversion rate up 0.578%, clicks up 0.370% and orders up 1.053%.
The paper also compares Strategy 5 with an expert-designed strategy. The agent’s final configuration improved GMV by another 2.150 percentage points and the gross profit measure by 2.941 points. Watch time, live GMV and platform orders also moved more favourably than under the expert version.
The paper does not disclose the number of users, traffic allocation, experiment duration, absolute baselines, confidence intervals or p-values for the production sequence. It says the system extracts statistical confidence from experiment records, but it does not publish that uncertainty for the final test.
The comparison also evaluates the complete agent-generated strategy. It does not isolate the contribution of the experience tree, retrieval method, language model, human review or individual parameter changes. The observed lift therefore belongs to the combined intervention.
The growth loop became the product
The transferable part of the experiment is not that every growth team needs an autonomous experimentation agent. Kuaishou had hundreds of previous strategies, a large production platform, mature metric infrastructure and enough traffic to run successive tests. Those conditions are difficult to reproduce.
The practical idea is smaller. Historical tests become more useful when their context, mechanism, constraints and failure modes are stored alongside the result. The structure allowed the Kuaishou A/B Agent to distinguish a superficially similar tactic from one that fit the current stage and guardrails.
The sequence also shows why a primary metric cannot make the decision alone. Strategies 1 and 2 improved shopping performance while weakening the surrounding platform. The next round was not simply a larger bet on the same setting. It changed the balance between the target and the constraints.
A comparable business could test this operating model before automating it. Record each experiment as a decision with context, intervention, primary outcome, guardrails and next step. Retrieve the closest precedents when designing the next test. Let an AI experimentation agent propose configurations, but require human approval and a predeclared set of measures that can stop the rollout.
Kuaishou’s paper ends with another unresolved constraint. Future work includes better requirement alignment, more accurate attribution and automated checks that a proposed strategy can be engineered safely. The experiment moved strategy selection closer to a closed loop, but the missing disclosures and retained human review show where that loop still opens.
Other platforms are testing similar ground. Taobao Live trained its AI shopping hosts to keep working when operators changed the rules around them, and ShareChat rebuilt its testing pipeline to reach significance on revenue metrics faster. Both point to the same shift as the Kuaishou A/B Agent, the experimentation process itself is becoming something companies test and optimise, not just a tool they use.
Algolia’s website experiment keeps a person between an AI-built variant and launch. It provides a useful contrast in where the experimentation system receives permission to act. Douyin’s incentive experiment shows why more activity can still produce less revenue.
| Experiment detail | What the reporting shows |
|---|---|
| Change | An AI agent proposed new experiment strategies, drawing on 310 past strategies. |
| Reported result | +4.8% GMV after five rounds, with platform guardrails held |
| Evidence | Company-authored paper, not independently replicated |
| Limits | The first two strategies lifted GMV but broke platform guardrails. |
FAQ
What did Kuaishou’s A/B Agent do?
It organised 310 historical strategies into a searchable tree, retrieved relevant precedents for a new request, generated a proposed configuration with GLM-5.1, and revised its next move from live experiment results. An engineer reviewed each strategy before it entered online testing.
What did the first two strategies reveal?
They raised GMV, by 1.123% and 2.984%, while watch time, live GMV and platform GMV declined. They show why automated testing needs predeclared guardrails, because a target metric win can hide damage elsewhere.
What were the final reported results?
Against production, Kuaishou reports GMV up 4.829%, a gross profit measure up 4.677%, order profit measure up 0.841%, conversion rate up 0.578%, clicks up 0.370% and orders up 1.053%.
Can another company copy this?
Not easily at the same scale, since Kuaishou had hundreds of past strategies and mature metric infrastructure. A smaller version is to record each experiment with its context, mechanism, guardrails and next step, and to require human approval for any AI proposed configuration.
Sources
- A/B Agent paper and author affiliations, version dated 5 August 2026
- Full technical report with the five-strategy sequence and production results
- Kuaishou company website
- Kuaishou Technology on LinkedIn
Spotted an error? Contact the desk. Our corrections policy explains how we handle corrections and updates.
Find the reported method, result and limitations of other experiments in the Growth Signal Index in IC Lab.