Elin Ledenvik
23 May 2024
Whether you’re looking to start A/B testing, or just want to find a tool that better matches your needs or budget, there are some considerations you need to take into account when you start looking for a new Experimentation platform. We’ve previously written about the general considerations in our post Which A/B testing tool is best – for you?
But one of the main reasons we get involved in helping with A/B testing tool assessments is technical issues or considerations. There might be questions around how a certain tool works with a certain framework, or a tool has already been implemented, but does not work the way it was expected to. This post is an overview of the most common technical environments that may require special considerations when looking for a new A/B testing tool.
A common misconception is that a web experimentation platform, or client-side tool, will work out-of-the-box regardless of the environment. But equally common is the notion that some environments will make testing impossible.
It’s mostly a matter of knowing which resources and implementation efforts you will need, and how you find a suitable tool.
When talking about different A/B testing tool types, the conversation often boils down to client-side versus server-side solutions. These terms can be a bit misleading, as client- and server-side usually refers to where the page or application is rendered, i.e. where the website’s code is converted into a visual display.
In this post I will use Web and Feature Experimentation to describe the different testing solutions instead, where Web experimentation solutions refer to tools that are implemented by adding a short javascript snippet to the page, and provide a visual editor where non-developer users can make changes without code, whereas Feature Experimentation is used for describing tools that are implemented in the backend code of the site, that needs developers to set up tests.
A Web Experimentation tool is implemented by adding a JavaScript snippet as high up in the code of each page as possible. When a page loads, the A/B testing snippet runs. It checks if there are any active experiments on the page, then controls if the visitor has a cookie with any previous assignments. If there is, the tool loads the same variant as the visitor has seen before.
If the visitor has not seen the test before, the tool randomly assigns a variant and applies the test changes to the page. The changes are made in the browser, using JavaScript that manipulates the existing content or injects new content.
The tool needs to be called every time it needs to do a reevaluation, that is every time the tool needs to run again and apply changes, which is usually at every page load. The content of the page also needs to be accessible with JavaScript when the tool applies the changes. This is why some of the below environments might mean trouble.
These are some environments that can affect your ability to use a Web experimentation tool, and what you can do about it.
A Single page application is a web application or website that dynamically rewrites the current page in the browser rather than loading entirely new pages from the server.
Since web experimentation tools are typically called on each new page load, this means the tool will not run again when the page is updated if there is no hard page reload. Then no experiments will be applied or removed.
Most web experimentation solutions have some built in support for SPAs. For example, many can automatically detect if the URL updates, even when there is no hard page reload. But if there is no change to the url when a new page or view is displayed, you need to check what support the tool provides. There are some tools which automatically observes the page for changes, while other has custom trigger conditions that you need to implement in order to activate or deactivate experiments on new views or pages.
SPAs can also affect the possibilities to apply changes through the visual editor. To avoid flickering (when the user sees a flash of the original content before seeing the variant), a web experimentation tool needs to be one of the very first things loaded on the page. At the same time, the elements that you want to change need to be present in the DOM when the tool runs through the active tests and applies the changes. In SPAs, it’s common to have content that loads dynamically after the tool has already tried to apply all test changes. In this case, you will see your changes in the visual editor, but not on the live page.
If the experimentation tool does not have functionality for detecting changes to the page, you might need a code editor and someone with JavaScript knowledge in order to be able to write tests.
To figure out if you need any special considerations related to SPA, ask yourself or your developers these questions:
What to do about it:
In order to make changes through JavaScript, you need to be able to target specific elements on the page. This is managed using Selectors.
Say that you have a call-to-action button on your page, the HTML could look like in the image below.
In this case, we can select the button using several different selectors.
If the selector of the element changes during the test, the changes won’t be applied to that element.
Some CSS frameworks use dynamically generated class names that can change when the application is rebuilt, or when there are updates. This will make them very unstable to use in a visual editor or client-side code editor. In this case, you will need to work together with your developers to add specific and reliable attributes or classes to the elements you want to change, so they remain the same during the test.
Example of classes in a site using a CSS framework with dynamically rendered classnames
To figure out if you need any special considerations related to CSS frameworks, ask yourself or your developers these questions:
What to do about it:
When using a Static Site Generator (like Gatsby), the page can be pre-built for example locally or on a server, and then stored and cached on a Content Delivery Network, a CDN. A CDN is a network of proxy servers that are geographically distributed in order to increase performance and availability. So instead of sending a request to the origin server, a request is sent to the proxy server that is geographically closest to the user. Since the page is already pre-built, and does not need to be “assembled” neither in the browser nor on the server, it loads very quickly.
An example of a Jamstack environment, where the site is uploaded through a repository instead of an origin server. The pages are rendered into static pages before being cached on a Content Delivery Network.
If parts of your site are built with a static site generator, the page might load so quickly it will be challenging to use a client-side tool without noticeable flickering, and you will need to do workarounds in order to be able to test on components on the page.
-----
Three reasons to avoid flickering:
----
One solution to this is to work together with your developers to hide the contents of a component that you want to test – in both original and variant – and then use JavaScript to inject the correct content to both variants using the A/B testing tool. If the component is set to a fixed height, there will be less content shifting, and there the component will have equal delay in both variants, having less impact on the test results.
Or look for a tool that does bucketing and changes on the CDN (on the “Edge”).
Note that this environment affects the choice of Feature Experimentation tools too. Many Feature Experimentation solutions require a call to a server to do bucketing, but in some cases this architecture might not have a server.
To figure out if you need any special considerations related to ShadowDOM, ask yourself or your developers these questions:
What to do about it:
The Shadow DOM is a web standard designed to encapsulate and scope CSS and JavaScript in web components. It means developers can create isolated DOM trees that cannot be affected by CSS or JavaScript from other parts of the web page, preventing accidental interference or collision between different parts of a web application. Imagine having parts of the page in a closed box – You can only paint the outside of the box, not the contents within.
Since Web experimentation tools uses Javascript and CSS to change the page, it can be hard or impossible to use a visual editor or client-side code editor to change anything within a Shadow DOM component, since it blocks the content from being affected by outside CSS and JavaScript.
Example of how the a ShadowDOM component can look in the rendered HTML code
The ShadowDOM can be “open” or “closed”. Open Shadow DOM means the elements inside it are still accessible using JavaScript methods, but it keeps the styling of the elements inside from bleeding into the main document. A closed ShadowDOM means the elements inside it are not even accessible using JavaScript.
Some tools have built in support for Shadow DOM, as long as it is “open”. And if you have JavaScript knowledge, you can use a code editor even if the visual editor wouldn’t work. But if a lot of the elements you will want to test are within a closed Shadow DOM, you won’t be able to use a visual editor or a client-side code editor, but will need to make the changes inside the code base instead.
To figure out if you need any special considerations related to ShadowDOM, ask yourself or your developers these questions:
What to do about it:
In most cases, A/B testing solutions exist; it’s just a matter of finding the right tool and having the right resources. By looking at the questions in this post, you can make sure you put your money and effort into a solution that works for your site. You can also be prepared for any additional implementation needs, or internal collaboration needed to get the testing done.
And if you ever need help, don’t hesitate to reach out.
Ready to accelerate your digital growth? Add your details and we’ll get back to you.
