When A/B testing you are interested in the test results. You want to primarily know whether the test 1) is significant, and 2) provides correct data. Our guide shows you how.
This is a follow up to our earlier blog post ” Integrate A/B testing & Analysis ” that helped you connect the tools Optimizely and Universal Analytics (Google Analytics). After you have followed the instructions from that post, started your A/B test and received data for a few days, you should verify that it looks real. And this blog post describes how!
Overview TL;DR
- Check that everyone that should be included in the A/B test is included
- Check goal conversions
- Separate sessions and users, compare between the tools
- Troubleshoot if the numbers do not match
Let’s get started!
1. Check that all relevant visitors are included
You should do this as soon as you have a few days worth of complete data for your A/B test. If there is something wrong with tracking for example for a specific mobile device or browser, you want to detect it early, before the whole period for the A/B test has passed. Otherwise it can distort the results of the test and you will need to start over. To ensure that all visitors who should be included in your A/B test really are included, you need to create two segments in Google Analytics.
Create a new segment that matches the criteria for A/B testing. Let’s say the A/B test runs on the home page, and that only visitors that pass through the home page are included in the test. Then the criterion is ”Filter Sessions Include > Page exactly matches ”/”. You should therefore get a segment matching visits all should have received a version assigned in the A/B test. Rename it to eg ”AB1-Should :
Now copy the segment and rename the new copy to, for example, “AB1”, as the segment will match all the visits that actually had a version assigned. Add an AND criterion below the existing criterion for Page we just sat. In the earlier blog post (”Integrate A/B-testning & Analysis”) we sat up a custom dimension for the A/B-test. You can now find the name you gave to your A/B custom dimension in the dropdown menu. Set it to match any A/B variations, for example “Filter Sessions Include > [Your A/B custom dimension] contains “optimizely”.
Now we have two segments, each containing only visit that passed the home page where the A/B test runs. One also shows just the visits that got a custom dimension assigned by Optimizely. Double-check that you have the date range set on the days when the entire A/B test has been active.
Go to the report Audience>Overview and check the report’s statistics for Sessions och Users. If everything is correct, the first segment “AB1-Should” should contain about as many Sessions och Users as the other segment “AB1”. If you want to be extra safe, you can also create a segment ”AB1-Missing” where you copy your ”AB1” segment, but instead of “Filter Sessions Include > [Your A/B custom dimension] contains “optimizely” set the opposite: “Filter Sessions Exclude > [Your A/B custom dimension] contains “optimizely”. This missing-filter should then be mutually exclusive with ”AB1” filter and they should together constitute as many visits/visitors as ”AB1-Should”.
Compare the number of Users you get from the segment ”AB1” to the total amount of Visitors reported by Optimizely for the same period.
If the numbers do not match, you may have problems with distribution in the A/B test. Check that Optimizely script is located inside and runs properly on all webpages. Double-check filtering for internal traffic so it is the same for both Google Analytics (GA) and Optimizely. Compare the two segments ”AB1-Should” and ”AB1” over various other dimensions of your GA reports to see if, for example, any traffic source, browser, landing page, geographic location and so on are over-represented in the difference between these two segments.
2. Check the quantity and distribution of each A/B version
Create a new segment for each version of the A/B test by repeating the process when you put up a segment that matched all versions. Rename the first segment to for example the ”AB1-Org” and set the matching ”Filter Sessions Include > [Your A/B custom dimension] to contain ”original.”
Create a segment for each of the variations in the same way.
Now compare the statistics from Audience > Overview and make sure that:
a) All A/B versions together have as many Sessions / Users as segment ”AB1”
b) All A/B versions have the number of Sessions / Users evenly distributed among themselves
If the two A/B versions together have more Sessions or Users than the segment ”AB1” has, it suggests that there is an overlap in the data and that visitors have been assigned more than one version over the period.
Check the date range and that you set your custom dimension on ”session” or ”visitor” and not on ”page view” when you set it up in the section under ”Admin>Property>Custom Definitions>Custom Dimensions” of the earlier blog post.
3. Check conversions on user level
Optimizely and Google Analytics define a conversion slightly differently. As a consequence, the conversion rates for the goals you set that both tools report by default within their respective interfaces are different.
Optimizely on user-level
– Visitors who Converted once or several times / Visitors = % Conversion Rate
Google Analytics, session-level:
– Sessions including one or several conversions / Sessions = % Conversion Rate
The same figures in Analytics in Optimizely
To be able to compare the number of conversions and the conversion rate within Google Analytics with the respective numbers in Optimizely you need to use segments and Audience>Overview-method that you used above to control the A/B test range and traffic per A / B version.
Start by copying the segment ”AB1-Org” and then add a criterion so that the visits (except that they should have been awarded the A/B version ”original”) also should match the criterion for the conversion goals you want to compare. For example, if you want to see the conversion for Purchases and have a URL for your receipt page set up as a goal in Optimizely, add an AND criterion for the same receipt page in the new copy of ”AB1-Org” and name it “AB1-Org-Conv”.
Then go back to Audience>Overview and the number of Users in the segment ”AB1-Org” should be consistent with the number of Visitors for the original within Optimizely. Likewise, the number of Users in the ”AB1-Org Conv” should be consistent with the number of Conversions within Optimizely. Repeat the process for each A/B version and compare with Optimizely.
Here you should be able to discover early on if any of the variations do not register conversions, or if there is a big difference between Optimizely and Google Analytics. For example, if there are less conversions in Optimizely, check that Optimizely code is on all the webpages. A common error is to forget to put Optimizely on a subdomain where an order or a purchase is completed. When the GA script is located on the subdomain, conversions appear in Google Analytics, but not in Optimizely.
You will also be able to use these segments as you set for tallying uplift and statistical significance later when A/B test has received more data and it is time to analyze it.
