There is already a great blog post on this topic published by Alhan Keser at Widerfunnel from 2015. The web moves fast and a lot has happened since 2015, e.g. Google Optimize has been released, Optimizely X has been released, the adoption of responsive web design is greater and the usage of mobile devices to surf the web is higher than ever before.
So, here is an updated list on how to solve flickering in your A/B tests (online experiments) and links to helpful resources.
Firstly – What is flickering?
Flickering is also referred to as Flash of original Content (or FOOC). It means that the user sees the original (control variation) before they see the modified version of your page.
Example on how flickering may appear in an A/B test
(Disclaimer: This might also be an intended but risky experiment)
Why does flickering happen in the first place?
Since most A/B testing tools are executed client side (in the user’s web browser) this issue is very common. This is how most A/B testing tools work:
Here is a diagram on how the Optimizely Classic snippet works (Source: Optimizely)
This diagram shows that the modifications we made within our A/B testing tool is injected to the website meanwhile the visitor loads the page. The injected code is then on-the-fly rearranging the elements so it corresponds to changes we made within the tool.
There is no publicly available information on how Google Optimize is executed, but there is a way to prevent flickering with the page-hiding snippet. More on this down below.
Why does it matter?
A study at Carleton University in Ottawa shows that impressions from a webpage are made in the first 50 milliseconds of viewing1. This means that the brain can make flash judgements almost as fast as the eye can take in the information. The study also shows that the first impression lasts, and this is commonly referred to as the Halo Effect.
In another study from 2014 at MIT, a team of neuroscientists found that the human brain can process entire images that the eye sees for as little as 13 milliseconds 2.
This means that:
- If a user sees a glimpse of the control variation (a flicker), unconscious or conscious, it’s plausible that the user acts on that first impression
- The user behaviour will then not be traceable to neither Control nor to the Variation.
TL;DR:
All experiment results that are conducted with flickering are unreliable. Basically we don’t know what kind of experiment we’re running and can’t draw any conclusions.
1Lindgaard G., Fernandes G. J., Dudek C. & Brown J. Behav. Inf. Technol., 25. 115 – 126 (2006)
2Anne Trafton, MIT News Office
8 solid ways to prevent flickering
1Put your snippet where it needs to go
The single most important thing is to place the A/B testing tool snippet where the provider tells you to, often this is in the top of the <head> element. Check your tools documentation for further details. We also just wrote a white paper with the latest recommendation on how to implement for Google Optimize.
Get our implementation guide for Optimize (can’t believe we’re giving this out)
Oops! We could not locate your form.
*Complete guide, 17 pages, must have for your implementer
2Reduce the size of your A/B testing tool snippet
Many tools loads all paused and draft experiments by default, this is to reduce the number of requests to the provider’s server. In your A/B testing tool there are probably settings to disable this. You can also go ahead and archive all your previous experiments to minimize the size of the script.
Also, many tools use jQuery to do DOM manipulation. If your website already uses jQuery, move it so it loads before your A/B testing tool snippet and then go ahead and exclude it from the tool. Read more about this in the jQuery appendix.
(Nothing in this bullet applies to Google Optimize, which handles the loading differently.)
Send the article to relevant person
3Reduce image sizes
Compress images before you upload them (to a CDN or your A/B-testing tool) with tools like ImageOptim. ImageOptim is an open source tool that compress images without losing quality.
Reduce the amount of images loaded on any given page. Perhaps the homepage does not need a 9-slide carousel, a recommended products carousel, a featured products carousel, and all the other graphics/bling.
Implement “lazy loading” of your images. This means only images onscreen are actually loaded. Images offscreen load as you scroll to them. Consider this for carousels and exceptionally long pages. There are plugins/tools to help accomplish this.
If you have a responsive website, use current best practices to load an appropriate image for that specific device. Check responsiveimages.org and css-tricks.com for further examples.
Mobile full, 640 x 270, uncompressed, 201 KB
Mobile optimized, 320 x 135, compressed, 45 KB, saved about 78% transfer size
Desktop full uncompressed, 1968 × 706, uncompressed, 1,3 MB
Desktop optimized, 984 × 353, compressed, 246 KB, saved about 81% transfer size
4Link your experiment code (jQuery)
For each selector $(‘.class’) the web browser needs to go through the whole webpage.
So instead of writing changes in your A/B-testing tool like this:
$('div').css('color', 'red');
$('div').css('font-size', '2em');
$('div').text('Hello World!');
Write your code like this instead:
$('div').css({'color', 'red', 'font-size', '2em'}).text('Hello World!');
5Use CSS instead of JS/jQuery
Often it’s much faster for the web browser to apply CSS than to run JavaScript/jQuery, so try to write your changes in CSS and apply them to your experiment.
However, don’t do this in Google Optimize, read about why in an upcoming blog post.
Get a notification when the Optimize post is published via our Newsletter
Note: newsletter is in Swedish.
6Cache selectors
To cache selectors means that you store an element as a variable so that you can re-use that element without your jQuery having to re-query the DOM in search of it every time you want to apply some code to that element. So if you plan on using a certain jQuery element more than once, cache it!
var container = $('.container');
container.css('color', 'red');
Is the same as:
$('.container').css('color', 'red');
Also, jQuery selectors by default perform their search within the entire DOM. But while defining the selector, you can pass the context, which limits the searching range for the jQuery selector. Read more about how to optimize jQuery selectors for better performance in this great blog post.
var $container = $("#container");
var $child = $(".child", $container);
When $child is called it will only search the context of the #container and not the whole DOM.
7Target elements by load order
Since many tools, e.g. Optimizely, executes the code from top to bottom, you should reorder your variation code so that it matches your website.
This chart is showing how Optimizely Classic evaluates every line of code.
(Source: Optimizely)
In the case above the execution of variation code will pause when the target element has not been loaded yet. So, if your first line of code targets the last element to load, your variation code will not be executed until after the page is fully loaded. Instead, target elements by the order in which they are loaded on the web page.
8Use page/element hiding snippet where needed
We can and we recommend that you ALWAYS write your own version of Google Optimize’s page-hiding snippet which will temporarily hide an <element> for X amount of time. How you do that is stated in the implementation guide we linked to above.
That’s our 8 recommendations to prevent flickering. If you’ve got questions or need help:
- Please write a comment below (your question might be relevant to others).
- Or, contact Andreas Gref, responsible for A/B testning at Conversionista.
Email A/B test hero Andreas
Good luck with your experiments!
Header image source: Optimizely
