Declaring a winner too early can lead to misleading decisions and lost revenue. Even if one variant looks better at first glance, the data may not be reliable yet.
GemX provides a Probability to win indicator inside the Analytics view to help you evaluate experiment performance. Use the checklist below to validate your results before applying the winning variant.
9 Checklists to Validate Your Test Results
#1. Confirm the Test Status
From your Shopify admin, go to the GemX Dashboard > Experiment, where you can see all test campaigns set up with GemX.
Hover over the experiment you want to check the results for, and click on the chart icon to open its analytics page.
Before reviewing metrics, make sure:
-
The test is currently running or has been properly ended.
-
The test was not recently restarted.
-
Traffic allocation was not manually changed during the test.
Restarting a campaign or adjusting the traffic split mid-test can affect the reliability. If major changes occurred, consider extending the test.
#2. Review Traffic Distribution
In the Analytics view, check:
-
Traffic split configuration (for example, 50/50)
-
Visitor volume per variant (Control vs Variant)
Make sure:
-
Both versions are receiving sufficient traffic
-
There is no extreme imbalance between variants
Minor differences in traffic are normal. However, if one version has significantly fewer visitors, the results may not be reliable yet.
#3. Validate Sample Size
Check the following metrics for each variant:
-
Total visitors
-
Total orders (or conversions)
Avoid declaring a winner if:
-
Order volume is very low
-
Only a small number of conversions separate the variants
A larger sample size increases result stability. If conversions are still limited, continue running the test until more data is collected.
#4. Evaluate “Probability to Win”
Click View all metrics to open the Performance Detail panel.
From here, you can review the Probability to win value for each variant.
This metric indicates how likely a variant is to outperform another based on collected data.
Before declaring a winner:
-
Ensure one variant has a clearly higher probability to win.
-
Avoid deciding if probabilities are close or fluctuating.
-
As a general guideline, wait until the leading variant reaches a high probability (for example, above 90–95%).
If probability is still unstable or changing significantly over time, allow the test to continue running.
#5. Check the Primary Goal Metric
In the Analytics panel, identify the metric marked as Goal (for example, Conversion rate).
Before applying a winner, confirm:
-
The leading variant performs better on the defined Goal metric.
-
The improvement is meaningful, not just marginal.
-
The higher “Probability to win” aligns with the Goal metric.
Do not select a winner based on secondary metrics (such as click-through rate or time on page) if they are not your defined campaign goals.
6. Review Revenue Impact
If revenue tracking is enabled, compare:
-
Revenue
-
Average order value (AOV)
-
Revenue per visitor
A variant may increase conversion rate but reduce overall revenue or AOV. Always consider the overall business impact before declaring a winner.
Prioritize revenue and profitability, not just percentage lift.
#7. Ensure Sufficient Test Duration
Before making a final decision, confirm:
-
The test has run long enough to cover normal traffic patterns
-
Both weekdays and weekends are included (if relevant to your store)
Avoid declaring a winner:
-
After only 1–2 days of data
-
During abnormal traffic spikes (for example, a sudden ad push or flash sale)
A full business cycle helps reduce bias.
#8. Check for External Influences
Review whether any major changes occurred during the test period, such as:
-
A new discount or promotion was launched mid-test
-
A theme or layout redesign
-
A significant increase in paid traffic budget
If major external factors affected the experiment, consider extending or restarting the test to maintain data integrity.
#9. Apply the Winning Template
Once all checklist items are validated:
1. Go to the Campaign
2. Review the winning template
3. Click the Make winner button
4. Continue hitting Choose winning template to let GemX process your decision.
Learn more: How to Apply the Winning Variation and End Your Test Safely
Then, you can monitoring performance after implementation
Post-implementation monitoring ensures the performance uplift remains consistent in real-world conditions.
When Is It Safe to Declare a Winner
You can confidently declare an A/B test winner when:
-
Traffic distribution is stable
-
Sample size and conversion volume are sufficient
-
One variant shows a clearly higher “Probability to win”
-
The defined Goal metric demonstrates meaningful improvement
-
Revenue impact is positive
-
The test duration covers normal business cycles
If any of these conditions are not met, allow the experiment to continue collecting data before making a final decision.
FAQs
