In CRO we work with data. It is our backbone. Our core.
But every so often you hear someone say “That is just a usability test, you don’t have any actual data”. A misconception that we need to address. It makes it harder for us to do what needs to be done and create those great experiences that both brings user value and convert.
Quantitative vs Qualitative
There are both quantitative and qualitative data. They are different in nature and used for different kinds of research but both of them are data points.
Quantitative is that data you usually see on dashboards and discuss in board rooms. It is numerical data or data that can be translated into some kind of statistic and is generalized from a larger population. If you look at the number of conversions, NPS or number of users that answered “I agree” in that last poll, you’re looking at quantitative data.
Examples of quantitative methods are digital analytics such as Adobe or Google Analytics, polls and surveys with closed answers (that means the user can only answer from a set of predetermined options). But the thing is that even an interview can be quantitative if you close the questions and don’t allow the user to elaborate.
Qualitative data is mainly used for explorative research. It digs into needs, opinions and motivations and explores those things that you can’t set a number on. You usually have a smaller sample size and the outcome is often in the form of descriptions of insights on users’ thoughts and behaviours.
Examples of qualitative methods are semi-structured or unstructured interviews, usability test, polls and surveys with open-ended answers. You can absolutely quantify insights from qualitative research but that is an additional study. Results from qualitative research with small populations cannot be scaled up to the entire population of, say, a user base. That will not be representative for a quantitative way. But the insight is still data.
What determines which kind of method to use?
Quantitative data is great if you want to answer questions like “How many users do X?”, follow trends over time and measure an effect or uplift. Doing quantitative research you get to know WHAT users do. Users bought this product, added to cart or clicked on this button. But WHY they did or didn’t do something, that is why you do qualitative research. To get the full scope of an issue you usually have to combine both kinds of data. You can’t have the WHAT without the WHY and vice versa.
You can’t have the WHAT without the WHY and vice versa
Going into research, you usually set up a research question. A way to scope an area of interest for the research study which going forward will help and guide you in which methods to use.
So, let’s take an example
If you want to explore what is the next step for your site or product your research question might be something like this:
What uncertainties do users have that prevents them from signing up for a trial period when they are browsing the website.
Digging into uncertainties means digging into the users’ feelings and behaviours. So a relevant set of methods would probably be:
- Poll. You can set up a poll and ask the users directly on the site with an open-ended question if they have any uncertainties and what those uncertainties are.
→ Outcome: What users say prevent them from completing the signup - Usability test. You set up scenarios that take users through your funnel and see if they get stuck anywhere and how they use the site.
→ Outcome: Both what users say prevents them, and also insights on a specific behaviour that users do that might distract them from continuing or indicate uncertainties - Funnel and event. By studying drops in the funnel and having event tracking for key interactions on the site you can learn about what the users do on the site.
→ Outcome: What users do, how they move on the site and key steps where you have issues
The thing here is that you can’t really remove any of these methods if you want an answer to the research question. You can replace one or most of them with other methods, but you still have to dig into the issue from different perspectives.
– Without the poll, you won’t get uncertainties from a larger set of users and might miss something important.
– Without the usability tests, you won’t be able to study what users actually do, those things they don’t know that they are doing.
– Without digging into the funnel and events you won’t know if there are certain areas that have greater issues than others or be able to conclude if some interactions are driving or killing the conversion. 
Triangulating data
To combine methods and different kinds of data is called triangulating. It basically means that you look at an issue from different perspectives to figure out the whole picture. It doesn’t need to be hard. Just keep on researching from different angles and see if the data still tells the same story regardless of how you look at it. Does a data point tell you something else than you expected? Then dig into why that is. It doesn’t mean that the study itself is wrong (beware of confirmation bias!) – it means that you don’t have the full picture yet. So keep on researching and triangulating!
Does a data point tell you something else than you expected? Then dig into why that is.
So to summarize
Have you tested everything but you don’t get any uplift? Perhaps you have all the tracking in place and you see that the problem seems to be a certain part of the funnel but whatever principle you try nothing changes?
Let go of the numbers for a second and try a different angle!
- Think about what you want to research. What is the issue at hand that you need to deep dive into
- Set up a plan for the research with both quantitative and qualitative methods that will give you the best chance of figuring out what the problem is.
- Do the research and summarize all data from your study. What does the data tell you? Do you have both the WHAT and the WHY?
From here you can take the project forward, do an A/B-test, redesign a page, change the copy on that product page. Whatever the issue is now you have shed enough light on the problem that you can take the next step.
Good luck with the research!
