How I Know When My Test is Statistically Significant

GemX uses a Bayesian method to help you evaluate experiment results. Instead of showing a traditional p-value or a simple “statistically significant / not significant” status, GemX estimates the probability that each variant is likely to win based on the selected winning metric.

You can find this in the Performance detail box, where GemX shows the Probability to win for each variant.

In most cases, a variant can be considered a likely winner when its Probability to win reaches around 95%. However, this number should not be the only factor you use before making a final decision.

Run Smarter A/B Testing for Your Shopify Store

GemX empowers Shopify merchants to test page variations, optimize funnels, and boost revenue lift.

Start Free Trial

How GemX Evaluates Experiment Results

GemX calculates the Probability to win using the winning metric selected for your experiment.

This winning metric can be either:

• Conversion Rate

• Revenue

The probability is calculated only based on the selected metric. For example, if your winning metric is Conversion Rate, GemX evaluates which variant is more likely to win based on conversion performance. If your winning metric is Revenue, GemX evaluates which variant is more likely to win based on revenue performance.

This is important because a variant may perform better on one metric but not on another. A version that improves Conversion Rate may not always generate higher Revenue, and a version with higher Revenue may not always have the highest Conversion Rate.

Learn more: How to Choose the Primary Metric (Winning Metric) for Your Experiment

What does “Probability to Win” Mean

Probability to win shows how likely a variant is to outperform the others based on the selected winning metric.

For example:

• If Variant B has a 95% Probability to win based on Conversion Rate, GemX estimates that Variant B is very likely to perform better than the Control for Conversion Rate.

• If Variant B has a 95% Probability to win based on Revenue, GemX estimates that Variant B is very likely to generate better revenue performance than the Control.

This gives you a clearer way to evaluate experiment results without relying on a traditional p-value.

When is a Test Statistically Significant in GemX

In GemX, you can generally treat a result as statistically confident when a variant reaches around 95% Probability to win.

However, this does not always mean the result is ready to act on immediately.

There are two types of confidence to consider:

Statistical Confidence

Statistical confidence tells you whether the observed difference is likely to be real instead of random noise.

In GemX, this is reflected through the Probability to win. The higher the probability, the more confident GemX is that one variant is outperforming the others based on the selected metric.

Decision Confidence

Decision confidence tells you whether the result is strong enough to make a business decision.

A test may show high probability, but you should still check whether the lift is meaningful, stable, and aligned with your original hypothesis before declaring a winner.

When is a Result Ready for Decision-Making?

A result is usually decision-ready when all of the following are true:

• The Probability to win has reached around 95%: This shows that one variant is highly likely to outperform the others based on the selected winning metric.

• The performance trend is stable across multiple days: Avoid declaring a winner based on a short spike. The winning variant should continue to perform well over several days.

• The lift is meaningful for your business: A small lift may be statistically measurable but not worth acting on. Check whether the improvement is large enough to impact conversion rate, revenue, or your business goal.

• The result aligns with your hypothesis: The winning result should make sense based on what you expected to learn from the experiment. If the result contradicts your hypothesis, review the data carefully before applying changes.

• The experiment has run long enough for your traffic level: A low-traffic store may need more time to collect reliable data. The test should also cover different shopping behaviors across weekdays and weekends.

• The winning metric matches your experiment goal: If your goal is to increase purchases, Conversion Rate may be the right metric. If your goal is to increase total sales value, Revenue may be more relevant.

Why a Test May NOT be Ready Even With High Probability to Win

A high Probability to win is a strong signal, but it should not be used in isolation.

You may need more time before making a decision if:

• The test has only been running for a short time

• One variant had an early spike but the trend is now flattening

• The traffic volume is still low

• The lift is very small

• The result changes significantly from day to day

• The winning metric does not match your actual business goal

• Revenue and Conversion Rate are telling different stories

For example, a variant may reach a high Probability to win for Conversion Rate, but the Revenue result may still be weak. In this case, you should review whether the test goal is to drive more purchases or generate higher revenue before declaring a winner.

What Happens If I Change the Winning Metric

You can change the winning metric after launching an experiment.

When you change the winning metric, GemX recalculates the Probability to win based on the newly selected metric.

For example, if your experiment was originally evaluated based on Conversion Rate and you switch the winning metric to Revenue, the Probability to win will be recalculated using revenue performance instead.

Because of this, it is best to choose your winning metric carefully before launching the experiment whenever possible.

Recommended Checklist Before Declaring a Winner

Before declaring a winner, check the following:

• The winning variant has reached around 95% Probability to win

• The result is stable across multiple days

• The experiment has covered both weekdays and weekends

• The lift is meaningful enough to act on

• The result supports your original hypothesis

• The selected winning metric matches your experiment goal

• There is enough traffic and data for a reliable comparison

• You have reviewed both Conversion Rate and Revenue if both are important to your decision

If most of these conditions are met, your test result is more likely to be decision-ready.

Learn more: How to Apply the Winning Variation and End Your Test Safely

Need more hands-on support?

Book an onboarding call to get guided setup and expert help.

Book A Call

FAQs

Does GemX use p-value to determine statistical significance?

No. GemX uses a Bayesian method instead of a traditional p-value. It shows the Probability to win for each variant based on the selected winning metric, helping you understand which version is more likely to perform better.

What Probability to win should I look for before declaring a winner?

A variant is generally considered a likely winner when its Probability to win reaches around 95%. However, you should also check whether the result is stable, meaningful, and aligned with your experiment goal before making a final decision.

Can a test be statistically confident but not decision-ready?

Yes. A result may show high Probability to win, but the lift may be too small, unstable, or not aligned with your business goal. That is why you should consider both statistical confidence and decision confidence before declaring a winner.

Does changing the winning metric affect Probability to win?

Yes. Probability to win is calculated based only on the selected winning metric. If you change the winning metric from Conversion Rate to Revenue, or the other way around, GemX recalculates the Probability to win using the new metric.

Should I stop a test as soon as one variant reaches 95% Probability to win?

Not always. Reaching around 95% is a strong signal, but you should still check whether the result has stayed consistent across multiple days, covered enough traffic, and produced a meaningful lift before stopping the test.

Realted Topics: 

Analytics

,

help-center