How Experiment Analytics Work in GemX

Experiment Analytics in GemX are designed to help you understand how users behave inside an experiment, not to replace Shopify Analytics or GA4.

This article explains how GemX collects, attributes, and displays experiment data, so you can read your test results with the right expectations.

Pro tip: Use this guide when you want to understand where the numbers come from before deciding how to analyze or act on them.

Key Takeaways

This article explains:

• How GemX tracks sessions, visitors, conversions, and revenue in experiments

• How traffic is assigned to Control and Variant

• How Journey Analysis and performance charts are built

This article does not explain:

• How to decide a winner

• How to optimize conversion rate

• How to compare experiments across time

For decision-making, see How to Analyze Experiment Results in GemX.

How GemX Collects Experiment Data

When a Session Is Counted

A session in Experiment Analytics is counted when a user enters an experiment scope.
This usually happens when the user lands on a page or template included in the experiment.

Sessions are experiment-scoped, meaning:

• Only traffic that actually experiences the experiment is counted

• Site-wide traffic outside the experiment is excluded

Because of this, session numbers in GemX may be lower than Shopify or GA4 totals.

Visitors vs Sessions

GemX distinguishes between:

• Sessions: individual experiment visits

• Visitors: unique users within the experiment context

A single visitor can generate multiple sessions. This is why conversion rate and revenue calculations are always tied to the experiment scope, not the entire store.

How Traffic Is Assigned to Variants

When a user first enters an experiment, GemX assigns them to:

• Control (version A), or

• Variant (version B)

This assignment is sticky for the duration of the experiment. Returning users will continue to see the same version they were originally assigned.

Because of natural traffic behavior:

• A 50/50 split does not guarantee equal session counts

• Small imbalances are expected and normal

How Conversions Are Tracked

What Counts as a Conversion

A conversion in GemX is recorded when an online store session completes an order.

Conversions are always attributed to:

• The variant the user was assigned to

• The session in which the conversion occurred

Revenue Attribution in Experiments

If revenue tracking is enabled, GemX attributes order revenue to:

• The variant the user saw during the experiment

• The session that led to the purchase

Important implications:

• One order does not always equal one session

• A variant can have fewer conversions but higher revenue per visitor

• Revenue metrics are always experiment-scoped

This explains why experiment revenue may differ slightly from Shopify reports.

How Journey Analysis Works

Journey Analysis visualizes how users move through key steps of the funnel during an experiment.

Each funnel step represents a meaningful page or action, such as:

• Entry landing page

• Product page

• Add to cart

• Checkout

• Purchase

Journey steps are not meant to show every page view. They focus on decision points, not full navigation paths.

How Drop-Off Rates Are Calculated

Drop-off percentages represent the proportion of users who do not move to the next step.

For example:

• If 200 users start at step 1, and 40 reach step 2,

• The transition rate is 20%, and the drop-off is 80%.

These percentages help identify where an experiment succeeds or fails, not whether it “wins".

Learn more: How to Use Journey Analysis to Identify the Drop-offs and Optimize Your Funnel

How Performance Over Time Charts Are Built

Performance over time charts aggregate experiment data by time period.

They are designed to answer one question: Is the experiment behavior stable over time?

Key things to know:

• Flat lines usually mean no traffic during that period

• Curves for Control and Variant should follow similar patterns

• Sudden spikes often reflect traffic changes, not variant quality

These charts are not meant to predict future growth.

How Key Metrics Are Calculated in Experiment Analytics

Conversion Rate

The conversion rate is calculated as:

This differs from Shopify Analytics, which often calculates CR using site-wide sessions.

As a result, GemX conversion rate may appear higher or lower depending on experiment scope.

Average Order Value (AOV)

AOV in Experiment Analytics is calculated using:

Because AOV is experiment-scoped:

• Changes in product mix can impact it significantly

• A higher conversion rate can still lead to lower AOV

This is expected behavior, not a data error.

Revenue per Visitor

Revenue per visitor combines volume and value:

This metric is often more reliable than conversion rate alone, especially for e-commerce experiments.

How Segmentation Data Is Generated

Visitor Type: New vs Returning

Visitor type is determined based on whether the user has previously visited the store.

• New visitor: who accesses your store for the first time after GemX installation

• Returning visitor: who comes back to your store after GemX installation

This classification is best used to explain behavior, not to declare winners. Small sample sizes can easily skew segmentation results.

Device and Traffic Source Breakdown

Device type and traffic source are derived from the session context.

These breakdowns help answer questions like:

• Does this variant work better on mobile?

• Is the result driven by paid traffic only?

They should be used to guide iteration, not final conclusions.

Why GemX Data May Differ From Google Analytics

Differences are expected because:

• Google Analytics track site-wide behavior

• Event timing and attribution rules differ

Small discrepancies do not indicate tracking issues.

Learn more: Why does My Testing Data in GemX Look Different from Others

How Experiment Analytics Should Be Used

Experiment Analytics should be used to:

• Understand behavior inside an experiment

• Compare variants within the same experiment

• Generate insights for follow-up tests

They should not be used to:

• Measure overall store performance

• Compare unrelated experiments

• Replace Google Analytics reporting

Need more hands-on support?

Book an onboarding call to get guided setup and expert help.

Book A Call

Related Articles

• How to Analyze Experiment Results in GemX

• Understanding Metrics and Session Views in GemX

• When to Stop an Experiment

Realted Topics: 

Analytics

,

help-center