When you A/B-test you’re interested in the result. You primarily want to know if 1) the test is significant and 2) outputs correct data. To integrate your analysis tool with your testing tool you get full backup to your A/B-tests.
We will split this tutorial in two parts, this is the first part which help you integrate Google Analytics to the A/B-testing tool Optimizely. Part two helps you read your measurements and data you’ve collected.
- Integrate Analytics and Optimizely and you can quality assure your A/B-test results.
- You save time and avoid A/B-testing fuck-ups due to bad data.
We will cover:
Why you want to view A/B-results in Universal Analytics
1. Installing Optimizely’s integration code
2. Define new Custom Dimension for your A/B Test 3. Setup your A/B test and activate in Optimizely 4. Starta A/B test and check that you get Custom Dimension-data Summary
Why you want to view A/B-results in Universal Analytics
There are several reasons to why data from your A/B-test aren’t collecting correctly as it should:
When it comes to split testing, the most dangerous mistakes are the ones you don’t realise you’re making.[…] This is just one of over 60 ways to break your AB test that I’ve discovered.
Craig Sullivan, @OptimiseOrDie CRO Experts Biggest A/B Testing Mistakes
To just have data in your A/B-testing tool disable you to make any conclusions of your A/B-test results. In contrast, if you send same data to your analysis tool you can easily compare the data and see where you have mismatches. In this first part we will look at how to setup Optimizely in a way so that your A/B-test data is imported to Universal Analytics as Custom Dimensions.
1. Installing Optimizely’s integration code
First you have to add an activation code to you existing Universal Analytics code. The code tells Optimizely to activate the integration and start send A/B-test data as Custom Dimensions. Universal If you run Universal Analytics script directly on your site:
a. Two rows of Optimizely-code in-between GA Create and GA Send Pageview
This is the code that you need to paste in between your existing Universal Analytics tracking code:
window.optimizely = window.optimizely || ; window.optimizely.push("activateUniversalAnalytics");
The code snippet are placed after the Universal Analytics call
ga('create'... but before the pageview that is sent to
ga('send', 'pageview'); See example below on row 17:
Recommended: Update your tracking code across your website so that you don’t have to redo this step when you’re later launch A/B-test on other parts of your site. Voila! You’ve now installed Optimizely’s integration code. Go to step 2 ”Setup Custom Dimensions” further down in this blogpost.
If you run Universal Analytics via Google Tag Manager:
a. New Custom HTML-tag in GTM
If you run Universal Analytics via GTM you’re probably not doing it through a Custom HTML-tag but through GTM’s template-based Universal Analytics-tag:
What we need to do to activate Optimizely so that A/B-test data is sent to Universal Analytics is simply adding two lines of code. The template Universal Analytics-tag in GTM doesn’t have any input fields that allow us to do so. Therefore we have to create a Custom HTML-tag where we paste the Universal Analytics code just like it would been pasted into the site directly.
How you setup the Custom HTML tag
- Go to Google Tag Manager and then ”Container Draft>Overview”. Click ”New>Tag”. Choose a name e.g.. ”Universal Analytics via Custom” and below ”Select Tag Type” choose ”Custom HTML Tag”.
- Go to Universal Analytics and your main property (or the property you want to send A/B-test data to) ”Admin>Property>TrackingInfo>TrackingCode”. Copy the tracking code and paste it in your Custom HTML Tag in GTM in the big field ”HTML”.
- When you have a Custom HTML-tag with your Universal Analytics tracking code we can easily modify the tracking code by also paste the two rows of code that is needed to integrate Optimizely: window.optimizely = window.optimizely || ; window.optimizely.push(”activateUniversalAnalytics”); The code above should be placed in the middle of the Universal Analytics-script, after (
ga('create'...) but before the pageview is sent (
ga('send', 'pageview');) . The modified Universal Analytics tracking code in your Custom HTML-tag should look like this:
Save and you should now have the Custom HTML-tag containing your Optimizely-integrated version of your UA tracking code parallel to your original Universal Analytics-tag: Note that your data will still be tracked as usual from the Universal Analytics tag until we also set a ”All pages”-rule for the Custom HTML-tag and choose to publish our changes.
- Attention! If you have settings like Display Advertising Features, Enhanced Link Attribution, Cross Domain Tracking, Enhanced Ecommerce, UserID, etc. you’ll also need to put those into the Custom HTML-tag. The format is the same as if you’ve would have run the UA-code directly on the site instead of using GTM. Copy and paste the code the code for the specific functions / settings you need from this site.
b. New property in Universal Universal Universal Universal(Recommended)
- This step is necessary and will take a while but is recommended if you want to be completely sure that you’re not losing any valuable data. A new property is important if you move your extra tracking as we mentioning above. Do like this: a.) Setup a new property in Universal Analytics via ”Admin”>[Property-dropdown]>CreateNewProperty>”. Rename it to e.g.. ”Custom HTML Test Property”. ”b.) Get the UA-code (UA-XXXXXXXX-Y) for the property under Admin>Property>TrackingInfo>TrackingCode” and paste it in your Custom HTML-tag instead of your U-code for your main property.
When you’re in the the next step publish your Custom HTML-tag it starts sending data to the test property while the main property is untouched. You should check the data in the test property under the first 48 hours that it matches the data in the main property (remember that Google Analytics has a time offset). You can also use the Real Time-reports to compare the data between your properties. You can then instead of closing all your firing rules for your UA-tag (se next step) just switch the UA-code so that the Custom HTML-tag starts sending it’s data to your main property and the Universal Analytics-tag start sending it’s data to your test property.
c. Move rule ”All pages” from the Universal Analytics-tag to the Custom HTML-tag and publish the changes
- Setup a new rule to fire ”All pages” for your Custom HTML-tag. Remove the same rule from your UA-tag. Publish.
Check so that your Universal Analytics-account still receive all data. Voila! You’ve now installed the Optimizely’s integration code.
2. Define new Custom Dimension for your A/B-test
To enable Optimizely to send A/B-test data to Universal Analytics you need to open a slot for the data to be collected in – this slot will be your Custom Dimension. You don’t need to repeat the previous step in this blogpost, when your integration code is in place you it’s active until you remove it and can be used with all your tests. What you must do for each A/B-test is to create a new Custom Dimension in Universal Analytics which is shown below:
- Go to your Universal Analytics and your main property or the property you want the data to be sent to (it has to be the same property as you modified with Optimizely’s integration code in the previous step). Go to ”Property>CustomDefinitions>CustomMetrics” and click ”New Custom Dimension”
- Choose a name for your Custom Dimension that is connected to the A/B-test e.g.. ”Optimizely1: Copy Startpage”. Choose scope for your Custom Dimension and since we want the value to be attached to the user even if they return to the site even if they don’t visit the pages that are being tested we set the scope ”User”. Click ”Create”.
- You’ve now created a Custom Dimension.
- Note the index for your Custom Dimension since you’ll be using it in the next step when you activate the integration in Optimizely:
3. Setup your A/B-test and activate the integration in Optimizely
To finalise the setup and start sending data you need to activate the integration when you setup the test in Optimizely: 1. Setup your A/B-test as you normally would in Optimizely. Go to Edit-mode in your experiment. 2. Under ”Options” go to ”Analytics Integration”: 3. Choose ”Enable: send experiment data to Universal Analytics”. In the dropdown-menu, use the index that you got from your Custom Dimension in the previous step of this guide. Click ”Apply”. Now the integration is active in Optimizely on this specific A/B-test.
4. Start the A/B-test and check that you get Custom Dimension-data
Now you’re ready to start your A/B-test! To check that you get correct data in Universal Analytics: 1. Start the A/B-test in Optimizely. 2. Check after a few hours in Universal Analytics if there is any Custom Dimension-data. The easiest way to come back and review the data is to setup a Custom Report. Go to the tab called ”Customisation” left of ”Admin”. Click ”+ New Custom Report”. Name your Custom Report to something that is unique to the test you’re running e.g.. ”Optimizely1: Copy Startpage”. Go to the blue dropdown-menu ”+ add metric” and add ”Session” and ”User”. If you want, you can always add more metrics such as defined goals,pages/visit,bounce rate, etc. Go to the green dropdown-menu ”+ add dimension” and you’ll find your Custom Dimension under the headline ”Custom Dimensions”, click ”add”. When you’re all done, save the report. 3. Open the report. If your integration works correctly and there’s been a few hours since you started the A/B-test you should now see some data that Optimizely generated in your new Custom Dimension for the A/B-test, one value for each variation in the A/B-test. It should look something similar:
Check that data is generated for all types of devices that is included in the test and that it is somewhat equal amounts of visitors on all test-variations. It’s normal that Universal Analytics and Optimizely differ in how much data they collect. The different variations in the test should have equal amount of visits.
Summary / Next steps
Congratulations! You now have two sources of data – Optimizely and Universal Analytics. You’ve implemented the integration code for Optimizely, added a Custom Dimension for your new A/B-test, activated the integration inside Optimizely, started the A/B-test and checked that you get correct data in both Optimizely and Universal Analytics. All you need to do now is wait. To really ensure that the data is correct you need to run the test and then compare the numbers between Universal Analytics and Optimizely. Our recommendation is to run the A/B-test over two whole sales cycles, a statistical significance of 95% or more and finally, a minimum of 100-200 conversions per variation. In the next step we look at what you can do when your A/B-test have started. We look at different ways of comparing the data between Universal Analytics and Optimizely and come to conclusions, tools to analyse the significance, things you need to know when looking at data, and more.
Do you have any questions regarding the integration? Leave a comment below and we’ll be sure to help you out.