Statistical significance is not a certification of truth. It is a measure of evidence strength. Getting that distinction wrong is one of the most expensive mistakes in digital marketing, and one of the most common.
What Problem Is Statistics Solving?
When you show variant A to 500 people and variant B to 500 people, you're not studying all your users. You're drawing a sample. Samples produce variation that has nothing to do with which variant is better, just the randomness of which 1,000 people happened to visit on those particular days. Statistical significance is the framework for distinguishing "this difference is probably real" from "this is probably noise."
The p-Value: What It Actually Means
A p-value of 0.05 means: if variant A and variant B perform identically in reality, there is a 5% chance you'd see a result this extreme by chance alone. It does not mean "there is a 95% chance variant B is better." It means the result is unlikely enough by chance that we're willing to call it significant, a subtle but critical distinction.
“Statistical significance is not a guarantee your winner will keep winning. It is a probabilistic statement about how much you should trust what you're seeing.”
Segmently Analytics
Confidence Intervals: The Metric That Matters More
A 95% confidence interval tells you the range of plausible true effect sizes. If your variant shows a 12% lift with a CI of [2%, 22%], the improvement is statistically real, but plan for something closer to 2% than 22%. Confidence intervals keep you honest about uncertainty that p-values alone paper over.
Three Mistakes That Invalidate Results
Peeking: stopping when you first see significance
Checking results daily and stopping the test at first significance inflates the probability of false positives dramatically. Pre-commit to a minimum runtime and target sample size before launch, and don't call the result until both criteria are met.
Running tests for too short a time
Even if you hit significance on day three, the experiment hasn't captured weekday/weekend differences or let novelty effects decay. Run experiments for a minimum of two full weekly cycles regardless of when significance is reached.
Testing too many metrics and calling any winner
Testing ten metrics and declaring a winner because one crossed the threshold is guaranteed to produce false victories. Pre-register your primary metric before launch and treat all others as exploratory.
A Practical Framework
- 1Pre-commit: write down your hypothesis, primary metric, and target sample size before launch.
- 2Don't peek: set a calendar reminder for your end date and look only once you've hit your targets.
- 3Read the confidence interval, not just significance: ask what the lower bound implies for the business case.
- 4Run for full weekly cycles: two weeks minimum for most websites.
- 5Document every result, including null results. A recorded null is more valuable than an undocumented winner.