Look at your analytics and compare your mobile and desktop conversion rates. For most websites they differ by 40% or more, meaning you're dealing with two meaningfully different user populations. Running a single unsegmented A/B test across both is not just suboptimal. It is methodologically wrong.
Two Audiences, One Website: A False Assumption
A mobile visitor navigating a long-form landing page with their thumb in portrait mode is having a fundamentally different experience from a desktop visitor with a full keyboard and time to read. They have different attention spans, interaction patterns, tolerances for friction, and often entirely different intent.
How These Audiences Differ
- Intent: desktop sessions are longer and more research-oriented; mobile sessions tend toward search, comparison, and point-of-decision moments.
- Interaction: desktop users click precisely; mobile users tap imprecisely. A CTA that's easy on desktop may require precision that thumbs find frustrating.
- Context: desktop = sustained focus. Mobile = interrupted, multitasking, and divided attention.
- Form tolerance: desktop users tolerate longer flows; mobile users abandon multi-step forms at significantly higher rates.
- Load sensitivity: mobile users on cellular connections bounce faster; performance optimizations disproportionately benefit mobile.
The Danger of Averaging Results Across Devices
The most dangerous scenario: desktop visitors prefer variant A, mobile visitors prefer variant B, but the aggregate calls A the winner because desktop traffic is larger. You've just shipped a change that actively harms your mobile audience. Without device-level segmentation you'll never see it.
“Aggregated A/B test results that ignore device type are not wrong; they are confidently wrong. The insight you need is in the segment you aren't looking at.”
Segmently
How to Segment Your Tests
Option 1: Device-targeted experiments
The cleanest approach is creating separate experiments for mobile and desktop visitors from the outset. Use your platform's device targeting to define each experiment's audience before launch. This doubles your experiment count but produces unambiguous, actionable results.
Option 2: Post-analysis segmentation
For tests already running, always analyze results by device before declaring a winner. If desktop and mobile are pulling in opposite directions, do not ship the aggregate "winner"; run device-specific follow-up tests first.
The Mobile-First Testing Mindset
For most digital businesses mobile now accounts for 55–70% of sessions. Testing with a device-agnostic mindset means optimizing for a minority and applying it to the majority. The businesses winning on mobile in 2025 started treating it as a distinct testable surface two or three years ago.