Experiment summary
- Sport Chek, a Canadian sporting goods retailer, tested a free shipping message on product pages after an earlier cart test lifted transactions 7.3%, according to agency Conversion.
- For about a month the product page variation held a lift of roughly 6%, but the final aggregate result was a 2.49% lift at 55% confidence.
- The curves converged on 29 July, when the free shipping threshold rose from $50 to $75.
- Split by period, the variation lifted conversion 6.56% at 96% confidence while the threshold was $50, then cut transactions by nearly 15% at 94% confidence at $75.
- The case study omits sample sizes, baselines and margin, and the split came after the breakpoint was found, so retailers should log every commercial change against the test timeline.
A product message lifted conversion until Sport Chek changed its free-shipping threshold. The same message then worked against it.
Sport Chek had reason to expect a free-shipping message to help. An earlier cart experiment had made the offer more prominent and told qualifying shoppers that they had earned it. The treatment increased transactions by 7.3 percent at 99.5 percent confidence, according to the agency running the test.
The Canadian sporting-goods retailer then moved the idea earlier in the purchase journey. It placed free-shipping information on product pages, where customers were still deciding whether to add an item to their cart.
For most of the experiment, the message appeared to be working. Near the end, its advantage collapsed. The test had not simply changed its mind. Sport Chek had changed the offer underneath it.
The cart test made qualification personal
Sport Chek sells sporting goods, footwear and apparel through stores and ecommerce. Its experiment was documented by Conversion, the agency that worked with the retailer and published the results.
A message that swings from a win to a loss is a trust signal, not just a wording problem. We look at that broader dynamic in our analysis of why online retail still needs a confidence layer. Retailers testing their shipping thresholds can run the numbers through our eCommerce ROI Dashboard.
The original cart displayed a general message that orders of $25 or more received free shipping. Sport Chek’s average order value exceeded $100, according to the case study, which meant most shoppers seeing the cart had probably qualified. The page still made them compare their basket with the threshold themselves.
The treatment made the calculation explicit. Qualifying shoppers saw a congratulatory message telling them they had free shipping. Shoppers below the threshold saw the exact amount they needed to add. The stated hypothesis was that greater prominence and confirmation of qualification would increase motivation and transactions.
The winning cart variation combined this message with changes intended to reduce distraction and improve clarity. The reported 7.3 percent transaction lift therefore cannot be assigned to the wording alone. The source gives a confidence level but does not disclose the sample size, duration, baseline conversion rate or allocation between the control and variations.
The product-page test lost its lift
The next experiment tested whether the same value proposition could help before shoppers reached the cart. In the control, free-shipping information appeared in a sitewide banner that visitors could close. It did not appear within the product page.
One variation reduced distractions and gave more emphasis to the add-to-cart button. A second built on those changes and added a prominent message offering free shipping on orders over $50. Because the second variation inherited the first variation’s design changes, the comparison did not isolate the shipping message from the surrounding page treatment.
For about a month, the free-shipping variation held a lift of roughly 6 percent over the control. The final aggregate result looked much weaker. Conversion reported a 2.49 percent lift with 55 percent confidence.
Michael St Laurent, then Conversion’s director of experimentation strategy, noticed that the curves had begun converging sharply on July 29. Normal events such as weekends or sales would usually affect all variations in similar ways. A sudden divergence suggested that something had changed for one treatment more than the others.
The team traced the break to Sport Chek’s shipping policy. On July 29, the qualifying threshold increased from $50 to $75. The variation displayed that number prominently. The control kept the offer inside the less visible sitewide banner.
The same message exposed two different offers
After separating the periods, Conversion reported that the product-page variation increased conversion by 6.56 percent at 96 percent confidence while the threshold was $50. Once the threshold moved to $75, the same variation reduced transactions by nearly 15 percent at 94 percent confidence.
The message had not become less visible or harder to understand. It had become better at drawing attention to a weaker offer. At $50, prominence reminded more shoppers about a benefit they could reach. At $75, the message told more shoppers that the benefit sat farther away.
A lower threshold can improve conversion while increasing shipping expense. A higher one can reduce orders while moving some baskets upward. The case study reports transactions, not contribution margin, shipping cost or average order value for each arm.
The split analysis also happened after the team found the July 29 breakpoint. The investigation has a clear operational explanation, but it was not described as a pre-planned segmentation. The reported confidence levels should therefore be read alongside the risk of finding a compelling pattern after examining the data.
Record commercial changes inside the test
The most transferable lesson concerns experiment operations. Pricing, promotions, inventory, delivery promises and eligibility rules can change while a test is running. If those events are not recorded against the experiment timeline, a treatment can appear to decay or recover without anyone understanding why.
A retailer testing a threshold message could use the existing page as the control and add a treatment that calculates qualification for each basket. Both arms should experience the same threshold, products, shipping regions and promotional periods. Any policy change should pause the test or start a separately labelled phase.
The primary metric should be contribution margin per eligible visitor rather than transaction rate alone. Conversion, average order value, items per order and shipping cost can explain how the treatment creates that margin. Cart abandonment and customer-service contacts should act as guardrails.
The decision threshold should require a pre-agreed improvement in contribution margin with acceptable uncertainty. If the business wants to compare two shipping thresholds, each threshold needs its randomised treatment at the same time. A before-and-after policy change leaves seasonality, promotions and traffic mix in the comparison.
Sport Chek’s test began as a question about where to place a free-shipping message. The policy change revealed the more useful question. Before making an offer louder, how many customers will hear it as a benefit?
A changing journey can also move a test result in the other direction. Sterling’s AI insurance sales agent was level with its control for two months after a quote journey change, then pulled ahead.
The same operational trap shows up elsewhere in this series. Bunzl found that a heavily-clicked homepage section was not the win it looked like, and Cann’s best-performing ad channel looked strong on a dashboard until a proper holdout test checked it. In each case, the number on the dashboard needed a second look before anyone trusted it.
Very’s offer-selection experiment extends the offer question to when a shopper should see an incentive at all.
| Experiment detail | What the reporting shows |
|---|---|
| Change | Kept the same message but raised the free-shipping minimum from $50 to $75. |
| Reported result | +6.56% at $50 threshold, -15% once threshold rose to $75 |
| Evidence | Agency case study, no sample size/baseline disclosed |
| Limits | The winning message turned into a loser once the price changed. |
FAQ
What did Sport Chek test?
After a cart test lifted transactions 7.3 percent, Sport Chek moved the free shipping message onto product pages. One variation emphasised the add to cart button and a second added a prominent offer of free shipping on orders over $50, against a control that only had a closable sitewide banner.
Why did a winning message become a loser?
On 29 July the qualifying threshold rose from $50 to $75, and the variation displayed that number prominently while the control kept it in a less visible banner. Conversion reported a 6.56 percent lift at $50 and a drop of nearly 15 percent at $75.
Does the test show that $50 was the best threshold?
No. A lower threshold can improve conversion while raising shipping cost, and the case study reports transactions but not contribution margin, shipping cost or average order value. The split analysis also happened after the team found the July 29 breakpoint.
How can a retailer protect a test from pricing or policy changes?
Record pricing, promotions, inventory and eligibility changes against the experiment timeline, and pause the test or start a labelled phase when one occurs. To compare two thresholds, randomise each at the same time, and judge the result on contribution margin per eligible visitor.
Source note Conversion ran the experiments and published the case study. The product-page tests took place in 2020, and the source was checked on September 21, 2026. It reports confidence levels and the date of the threshold change but omits sample sizes, traffic allocation, absolute conversion rates and financial outcomes.
Spotted an error? Contact the desk. Our corrections policy explains how we handle corrections and updates.
Find the reported method, result and limitations of other experiments in the Growth Signal Index in IC Lab.
