Elin Ledenvik
12 May 2020
One of our 21 most common A/B testing questions is: Which A/B testing tool do you recommend? There are already several articles out there with titles like: “The 23,5 best A/B testing tools in 2020” and although they can surely be both helpful and informative, it doesn’t always make it easier to make a choice. Comparisons in the form of checklists of features included in different tools might not be what you really need either. In the end, is it really the tool with the most features you should go for?
In this post you won’t get a comparison of all the tools we’ve tried, culminating in a revelation of which one is The Best Tool™. It will rather tell you about how to think when choosing a tool that will be most suited to your needs.
But first, what is an A/B testing tool, and what are the differences anyway?
An A/B testing tool should do three things:
But there are maybe 80 + tools out there, what are the differences then?
Most often, there are no major differences. Most tools work in the same way and offer somewhat similar features. What differs are often things like price, statistics engine and additional features.
One of the most important aspects when it comes to how the tool will be experienced and used is the perceived ease of use and intuitiveness of the UI. But this is also one of the most subjective aspects since it depends on the users’ preferences and previous experience. Some tools might feel more intuitive for people who want to use the tool without knowing any code, while other tools might be more suited for more technical users.
The most important thing is not which tool you choose, but that you actually get one and start experimenting. But if you want to feel like you’ve made a good assessment before you go ahead and buy an expensive tool, you can follow the steps below.
First of all, you need to map out your specific requirements and define which features you actually need. These are some questions you could ask yourself:
Try to summarize your general plan for experimentation. This will help give you an overview of the basic functionalities you will need a tool to have.
*Server-side testing
A/B testing tools most commonly make changes client-side, i.e. the changes are made in the visitor’s browser through Javascript. Some tools also provide a solution to run tests server-side, which give you the possibility to change content that needs to be changed on the server before the page is loaded in the browser, for example, product-level changes. Some tools also provide solutions for experimenting on Mobile Applications. Server-side testing has some major advantages like better site performance and eliminating the notorious “flicker effect”.
Look at everything concerning the people involved in experimentation and needs associated with this. This will give you an overview of the administrative capabilities you need as well as the level of technicality you want.
Are there other aspects in regards to the organization and current setup that will affect your decision?
Some tools offer personalization functionalities, others have tools for planning and ideation and so on. Some platforms also include other features such as heatmaps, scroll maps, session recordings, but you could also use a separate tool for these functions.
Do you want other features than just classic A/B testing? Which ones?
Look at what integrations that need to be possible
Do you need some specific API?
What level of support do you think you will need and what kind of support do you prefer?
Are documentation and guides important?
Try to estimate the amount of traffic you will be running your experiments on. How much testing do you want to do ideally, and what is reasonable to think you will actually achieve? If you have done testing before, you might look at the average number of users per experiment and multiply that with the estimated number of tests you plan on running during a year.
Estimated traffic = Average number of users/test * Estimated number of experiments/year
Do you have a budget? Try to make an estimation of what you could pay for a tool and still get a return on investment, given the amount of testing you will be doing.
On average, 3/10 tests will be winners but remember you will also be saving money on not implementing ideas that lower your conversion rate. Will the money you earn/save from experiments be enough to cover the cost of the tool?
By answering the questions above, you should be able to make a list of requirements. Dividing it into need-to-haves and nice-to-haves is recommended since not all functions and features will be equally important. Don’t get stuck wanting a bunch of cool features you’re not sure you’re actually going to use.
"Don’t get stuck wanting a bunch of cool features you’re not sure you’re actually going to use."
When you have a list of requirements, you can start looking at checklists for different tools to see which tools that have the features and functions you need.
If you need some inspiration, here are some common A/B testing tools:
No matter which tool you choose, it is common to be frustrated with it during the implementation phase. It is (unfortunately) part of the implementation process having to wait for Legal to approve the tool, fighting to get implementation prioritized in development, trying to get in touch with someone at the tool to solve site-specific implementation problems, getting the tracking in place, cursing because the Google Analytics integration won’t work etc. Keep breathing. Take the opportunity to start meditating and see this as an opportunity for personal growth. Dream of all the tests you are going to run when everything finally works. Remember, you are not alone!
"Dream of all the tests you are going to run when everything finally works. Remember, you are not alone!"
Ok, maybe I’m exaggerating, but be prepared for a bit of friction.
If you don’t already, now is the time to make sure you have a process in place in order to run continuous testing and for prioritizing the ideas that have the most potential.
Blah, couldn’t you just tell me which tool I should choose instead?
The answer is that it doesn’t really matter which tool you choose, the important thing is that the tool gets used, and how. That’s why it’s important to start by looking at your specific needs. It’s better to start small and get going with your experimentation, than buying the most expensive tool there is but then leave it lying around unused in a digital corner.
Try to choose a tool that you think you and your team will find easy to use, rather than the one with the most advanced features. Above everything else, make sure you get going with the testing and get a good process in place. After all, it’s the process, not the tool, that is most important!