Hero Image A/B testing tool

Whether you’re looking to start A/B testing, or just want to find a tool that better matches your needs or budget, there are some considerations you need to take into account when you start looking for a new Experimentation platform. We’ve previously written about the general considerations in our post Which A/B testing tool is best – for you?

But one of the main reasons we get involved in helping with A/B testing tool assessments is technical issues or considerations. There might be questions around how a certain tool works with a certain framework, or a tool has already been implemented, but does not work the way it was expected to. This post is an overview of the most common technical environments that may require special considerations when looking for a new A/B testing tool.

I want a client-side solution, do I need to care about the tech?

A common misconception is that a web experimentation platform, or client-side tool, will work out-of-the-box regardless of the environment. But equally common is the notion that some environments will make testing impossible.

It’s mostly a matter of knowing which resources and implementation efforts you will need, and how you find a suitable tool.

A note on Client vs. Server-side

When talking about different A/B testing tool types, the conversation often boils down to client-side versus server-side solutions. These terms can be a bit misleading, as client- and server-side usually refers to where the page or application is rendered, i.e. where the website’s code is converted into a visual display. 

In this post I will use Web and Feature Experimentation to describe the different testing solutions instead, where Web experimentation solutions refer to tools that are implemented by adding a short javascript snippet to the page, and provide a visual editor where non-developer users can make changes without code, whereas Feature Experimentation is used for describing tools that are implemented in the backend code of the site, that needs developers to set up tests.

How A/B test tools for Web Experimentation work

A Web Experimentation tool is implemented by adding a JavaScript snippet as high up in the code of each page as possible. When a page loads, the A/B testing snippet runs. It checks if there are any active experiments on the page, then controls if the visitor has a cookie with any previous assignments. If there is, the tool loads the same variant as the visitor has seen before.

If the visitor has not seen the test before, the tool randomly assigns a variant and applies the test changes to the page. The changes are made in the browser, using JavaScript that manipulates the existing content or injects new content.

The tool needs to be called every time it needs to do a reevaluation, that is every time the tool needs to run again and apply changes, which is usually at every page load. The content of the page also needs to be accessible with JavaScript when the tool applies the changes. This is why some of the below environments might mean trouble.

Client side testing image

Four environments that might be troublesome

These are some environments that can affect your ability to use a Web experimentation tool, and what you can do about it.

  • Single Page Applications (SPA)
  • CSS frameworks with dynamic naming
  • Static site generator / CDN caching
  • Shadow DOM

Single Page Applications (SPA)

A Single page application is a web application or website that dynamically rewrites the current page in the browser rather than loading entirely new pages from the server.

Since web experimentation tools are typically called on each new page load, this means the tool will not run again when the page is updated if there is no hard page reload. Then no experiments will be applied or removed.

Flow of client-side rendering

Most web experimentation solutions have some built in support for SPAs. For example, many can automatically detect if the URL updates, even when there is no hard page reload. But if there is no change to the url when a new page or view is displayed, you need to check what support the tool provides. There are some tools which automatically observes the page for changes, while other has custom trigger conditions that you need to implement in order to activate or deactivate experiments on new views or pages.

SPAs can also affect the possibilities to apply changes through the visual editor. To avoid flickering (when the user sees a flash of the original content before seeing the variant), a web experimentation tool needs to be one of the very first things loaded on the page. At the same time, the elements that you want to change need to be present in the DOM when the tool runs through the active tests and applies the changes. In SPAs, it’s common to have content that loads dynamically after the tool has already tried to apply all test changes. In this case, you will see your changes in the visual editor, but not on the live page.

If the experimentation tool does not have functionality for detecting changes to the page, you might need a code editor and someone with JavaScript knowledge in order to be able to write tests.

To figure out if you need any special considerations related to SPA, ask yourself or your developers these questions:

  • Is your site a Single Page Application?
  • Which parts of the site are built as an SPA?
  • Do you have virtual page views, and are there places where there’s no url updates even though a new view/page is shown?

What to do about it:

  • Make sure your tool has support for SPAs, either out-of-the-box functionality for observing DOM changes or adding custom trigger events.
  • Make sure the tool has a client-side code editor so you can write and add JavaScript if you can’t use the visual editor.

CSS frameworks with dynamic naming

In order to make changes through JavaScript, you need to be able to target specific elements on the page. This is managed using Selectors.

Say that you have a call-to-action button on your page, the HTML could look like in the image below.

Button example

In this case, we can select the button using several different selectors.

  • The element type, a, would target all links on the page unless we are more specific. 
  • The class, primary-button, would target all elements which we assigned that class, in this case all buttons that we want to style as a call-to-action button.
  • The ID is unique to a single element, in this case the specific button with the ID “contact-button”.

If the selector of the element changes during the test, the changes won’t be applied to that element.

Some CSS frameworks use dynamically generated class names that can change when the application is rebuilt, or when there are updates. This will make them very unstable to use in a visual editor or client-side code editor. In this case, you will need to work together with your developers to add specific and reliable attributes or classes to the elements you want to change, so they remain the same during the test.

CSS framework example

Example of classes in a site using a CSS framework with dynamically rendered classnames

To figure out if you need any special considerations related to CSS frameworks, ask yourself or your developers these questions:

  • Are you using this type of CSS framework? (For example Styled components)
  • Do the class names change with each rebuild?
  • Do you have any other considerations that will make targeting specific elements by class name hard?

What to do about it:

  • Work together with your developers to add specific and reliable attributes or classes that won’t change during the test.

Static site generator / CDN caching

When using a Static Site Generator (like Gatsby), the page can be pre-built for example locally or on a server, and then stored and cached on a Content Delivery Network, a CDN. A CDN is a network of proxy servers that are geographically distributed in order to increase performance and availability. So instead of sending a request to the origin server, a request is sent to the proxy server that is geographically closest to the user. Since the page is already pre-built, and does not need to be “assembled” neither in the browser nor on the server, it loads very quickly.

A jamstack example

An example of a Jamstack environment, where the site is uploaded through a repository instead of an origin server. The pages are rendered into static pages before being cached on a Content Delivery Network.

If parts of your site are built with a static site generator, the page might load so quickly it will be challenging to use a client-side tool without noticeable flickering, and you will need to do workarounds in order to be able to test on components on the page.

-----

Three reasons to avoid flickering:

  • It can lead to bad UX and make the page look broken
  • If the visitor has time to see the original before the variant, that can affect their behavior and thus the test results
  • Flickering might draw attention to the element, inadvertently giving it more attention in the variant

----

One solution to this is to work together with your developers to hide the contents of a component that you want to test – in both original and variant – and then use JavaScript to inject the correct content to both variants using the A/B testing tool. If the component is set to a fixed height, there will be less content shifting, and there the component will have equal delay in both variants, having less impact on the test results.

Or look for a tool that does bucketing and changes on the CDN (on the “Edge”).

Note that this environment affects the choice of Feature Experimentation tools too. Many Feature Experimentation solutions require a call to a server to do bucketing, but in some cases this architecture might not have a server.

To figure out if you need any special considerations related to ShadowDOM, ask yourself or your developers these questions:

  • Are any parts of the site built with a Static Site Generator?
  • What parts of the site are static/cached on a CDN?
  • Which type of content is cached? Whole pages or just parts?
  • Feature Experimentation: Do you use an origin server?

What to do about it:

  • Look at what options your A/B test vendor is offering, but talk to your developers to see what is viable.
  • Work together with your developers to hide content of the components you want to test, both for original and variant. Set a fixed height in order to limit content shifting. Then load the right content depending on the variant that the user is assigned.
  • Consider using dynamic content where you need to be able to A/B test.
  • Look for a solution that does bucketing on the Edge

Shadow DOM

The Shadow DOM is a web standard designed to encapsulate and scope CSS and JavaScript in web components. It means developers can create isolated DOM trees that cannot be affected by CSS or JavaScript from other parts of the web page, preventing accidental interference or collision between different parts of a web application. Imagine having parts of the page in a closed box – You can only paint the outside of the box, not the contents within.

Since Web experimentation tools uses Javascript and CSS to change the page, it can be hard or impossible to use a visual editor or client-side code editor to change anything within a Shadow DOM component, since it blocks the content from being affected by outside CSS and JavaScript.

Example of shadow root element in DOM tree

Example of how the a ShadowDOM component can look in the rendered HTML code

The ShadowDOM can be “open” or “closed”. Open Shadow DOM means the elements inside it are still accessible using JavaScript methods, but it keeps the styling of the elements inside from bleeding into the main document. A closed ShadowDOM means the elements inside it are not even accessible using JavaScript.

Some tools have built in support for Shadow DOM, as long as it is “open”. And if you have JavaScript knowledge, you can use a code editor even if the visual editor wouldn’t work. But if a lot of the elements you will want to test are within a closed Shadow DOM, you won’t be able to use a visual editor or a client-side code editor, but will need to make the changes inside the code base instead.

To figure out if you need any special considerations related to ShadowDOM, ask yourself or your developers these questions:

  • Are you using Shadow DOM?
  • Which parts of the page? Is it limited to certain components or bigger parts of the site?
  • Closed or open Shadow DOM? If it’s open, some tools have support for making the visual editor work. If it’s closed, we won’t be able to use a client-side tool within these components/areas of the site without backend dev.

What to do about it:

  • Check if the A/B testing tool has support for Shadow DOM
  • Use the code editor instead of the visual editor if you have resources with JavaScript knowledge.
  • If you are using closed Shadow DOM in big parts of the site, consider doing feature testing instead
  • Let developers write two versions of the component, then use the A/B testing tool to add an attribute of the wrapping container in order to load different variants

There’s always a solution

In most cases, A/B testing solutions exist; it’s just a matter of finding the right tool and having the right resources. By looking at the questions in this post, you can make sure you put your money and effort into a solution that works for your site. You can also be prepared for any additional implementation needs, or internal collaboration needed to get the testing done.

And if you ever need help, don’t hesitate to reach out.

Reach out today, grow your business tomorrow

Ready to accelerate your digital growth? Add your details and we’ll get back to you.