One of our 21 most common A/B testing questions is: Which A/B testing tool do you recommend? There are already several articles out there with titles like: “The 23,5 best A/B testing tools in 2020” and although they can surely be both helpful and informative, it doesn’t always make it easier to make a choice. Comparisons in the form of checklists of features included in different tools might not be what you really need either. In the end, is it really the tool with the most features you should go for?
In this post you won’t get a comparison of all the tools we’ve tried, culminating in a revelation of which one is The Best Tool™. It will rather tell you about how to think when choosing a tool that will be most suited to your needs.
But first, what is an A/B testing tool, and what are the differences anyway?
What does an A/B testing tool do?
An A/B testing tool should do three things:
- Randomly send visitors to different variations and make sure they stay in the same variation throughout the experiment
- Customize the customer experience by e.g manipulating content or redirecting the visitor to another page
- Track behaviour and result
But there are maybe 80 + tools out there, what are the differences then?
Most often, there are no major differences. Most tools work in the same way and offer somewhat similar features. What differs are often things like price, statistics engine and additional features.
One of the most important aspects when it comes to how the tool will be experienced and used is the perceived ease of use and intuitiveness of the UI. But this is also one of the most subjective aspects since it depends on the users’ preferences and previous experience. Some tools might feel more intuitive for people who want to use the tool without knowing any code, while other tools might be more suited for more technical users.
So how do I choose?
The most important thing is not which tool you choose, but that you actually get one and start experimenting. But if you want to feel like you’ve made a good assessment before you go ahead and buy an expensive tool, you can follow the steps below.
Define your prerequisites and requirements
First of all, you need to map out your specific requirements and define which features you actually need. These are some questions you could ask yourself:
1. What do you want to test?
Try to summarize your general plan for experimentation. This will help give you an overview of the basic functionalities you will need a tool to have.
- Where do you want to test? Will you only focus on your website, or also on a mobile application?
- What type of changes will you be making? Do you need a server-side solution?*
- What types of tests do you need to be able to run? A/B/n, multivariate, split tests?
- Do you want to test one or more sites?
- Do you have specific targeting or tracking needs? Specific audiences?
*Server-side testing
A/B testing tools most commonly make changes client-side, i.e. the changes are made in the visitor’s browser through Javascript. Some tools also provide a solution to run tests server-side, which give you the possibility to change content that needs to be changed on the server before the page is loaded in the browser, for example, product-level changes. Some tools also provide solutions for experimenting on Mobile Applications. Server-side testing has some major advantages like better site performance and eliminating the notorious “flicker effect”.
2. Who will be involved?
Look at everything concerning the people involved in experimentation and needs associated with this. This will give you an overview of the administrative capabilities you need as well as the level of technicality you want.
- How many collaborators do you need the tool to allow?
- Do you want to be able to assign different roles and access levels?
- Who will do the test setup in the tool and what are their technical skills and previous experience?
- Who will develop the tests and what availability do they have? To what extent will they be able to help? If you have limited development resources, maybe server-side testing won’t be an option at this point, for example.
3. What other technical prerequisites and limitations do you have?
Are there other aspects in regards to the organization and current setup that will affect your decision?
- Do you already have another tool implemented? How hard is it to get a new tool implemented? Will it be cheaper/easier to keep the tool you already have?
- Does the legal department or IT have to be involved in the choice?
- What setup will be required?
4. Which features do you expect an A/B testing tool to have?
Some tools offer personalization functionalities, others have tools for planning and ideation and so on. Some platforms also include other features such as heatmaps, scroll maps, session recordings, but you could also use a separate tool for these functions.
Do you want other features than just classic A/B testing? Which ones?
5. Which integrations should be possible?
Look at what integrations that need to be possible
- How will you do your analysis? Do you use Google Analytics, Adobe Analytics or some other tool? Maybe you also want to be able to export data?
- Do you want to be able to import third party data into the A/B testing tool, for example customer segments from a database or data warehouse?
- What other tools do you use that you might want to integrate?
- Do you need some specific API?
6. What will your need for support look like?
What level of support do you think you will need and what kind of support do you prefer?
- Do you want to be able to call, email or chat?
- Do you want to have a dedicated contact person from the platform vendor?
- Are documentation and guides important?
7. Estimate traffic and budget
Try to estimate the amount of traffic you will be running your experiments on. How much testing do you want to do ideally, and what is reasonable to think you will actually achieve? If you have done testing before, you might look at the average number of users per experiment and multiply that with the estimated number of tests you plan on running during a year.
Do you have a budget? Try to make an estimation of what you could pay for a tool and still get a return on investment, given the amount of testing you will be doing.
On average, 3/10 tests will be winners but remember you will also be saving money on not implementing ideas that lower your conversion rate. Will the money you earn/save from experiments be enough to cover the cost of the tool?
Make a list of requirements
By answering the questions above, you should be able to make a list of requirements. Dividing it into need-to-haves and nice-to-haves is recommended since not all functions and features will be equally important. Don’t get stuck wanting a bunch of cool features you’re not sure you’re actually going to use.
Don’t get stuck wanting a bunch of cool features you’re not sure you’re actually going to use.
When you have a list of requirements, you can start looking at checklists for different tools to see which tools that have the features and functions you need.
If you need some inspiration, here are some common A/B testing tools:
- AB Tasty
- Adobe Target
- Convert
- Google Optimize
- Kameleoon
- Maxymizer
- Monetate
- Omniconvert
- Optimizely
- VWO
- Webtrends Optimize
One last word of advise
No matter which tool you choose, it is common to be frustrated with it during the implementation phase. It is (unfortunately) part of the implementation process having to wait for Legal to approve the tool, fighting to get implementation prioritized in development, trying to get in touch with someone at the tool to solve site-specific implementation problems, getting the tracking in place, cursing because the Google Analytics integration won’t work etc. Keep breathing. Take the opportunity to start meditating and see this as an opportunity for personal growth. Dream of all the tests you are going to run when everything finally works. Remember, you are not alone!
Dream of all the tests you are going to run when everything finally works. Remember, you are not alone!
Ok, maybe I’m exaggerating, but be prepared for a bit of friction.
If you don’t already, now is the time to make sure you have a process in place in order to run continuous testing and for prioritizing the ideas that have the most potential.
Conclusion
Blah, couldn’t you just tell me which tool I should choose instead?
The answer is that it doesn’t really matter which tool you choose, the important thing is that the tool gets used, and how. That’s why it’s important to start by looking at your specific needs. It’s better to start small and get going with your experimentation, than buying the most expensive tool there is but then leave it lying around unused in a digital corner.
Try to choose a tool that you think you and your team will find easy to use, rather than the one with the most advanced features. Above everything else, make sure you get going with the testing and get a good process in place. After all, it’s the process, not the tool, that is most important!
