I’ve seen the AI-future of CRO. And it’s bright!

In the past, CRO relied on human intuition, expertise, and data analysis. However, with the rise of artificial intelligence (AI), the process of CRO is rapidly changing. In this blog post, we will explore how AI will affect the practice of conversion rate optimisation.

This was the first phrase ChatGPT spat out when I asked it to write a blog post about how AI will affect the practice of CRO. I know, asking ChatGPT to produce content for you in this way doesn’t really work. And yes I know, this is almost becoming a cliché, doing little to no work in prompting and being amazed by getting a seemingly human response. So bear with me, you don’t know where I am going with this yet.

So, back to the response. Interestingly ChatGPT “thinks” that the process used to rely on “human intuition, expertise, and data analysis”, but AI will change this. Let’s examine if human intuition and expertise will be replaced, and in that case with what?!

In order to get a better picture of the challenge, let’s break down the process of CRO into four simple and generic steps. Then let’s try to find out how AI, and in particular generative AI, will affect the work of CRO practitioners in each of those steps.

A very general CRO Process

The CRO process generally consists of four steps: 

1. Insights – Understanding user behaviour through a combination of quantitative and qualitative data.
2. Ideation – Developing hypotheses for improvements aimed at important KPIs.
3. Experiment – Validating the hypothesis through a series of controlled experiments, where the winner is determined through a statistical, scientific method.
4. Reporting – Presenting the findings and highlighting suggested actions.

If you use a method or a framework for your CRO work, it probably differs a little, but I would also guess that it’s not too far from this general method.

What job should the AI do for us then?

So, we want AI to help us with our CRO process. But then – what is it that the AI does? Or in particular – what is it that generative AI is really good at, where it could complement our human efforts or maybe replace them? Let’s break this down too.

I’m not getting lazy here, it’s just part of an underlying experiment. So let’s see what ChatGPT can do for me for that breakdown. This is what I got:

“Generative AI is a powerful tool that can assist analysts in a variety of ways, from gathering and structuring large data sets to creating polished presentations and transforming information to suit a particular audience. Let’s look at each of those capabilities in more detail:”

1. Gathering, structuring, and summarising:

Generative AI will assist analysts in gathering, structuring, and summarising large data sets by using machine learning algorithms to identify patterns and trends. This allows analysts to quickly identify key insights and make data-driven decisions. Generative AI can also help to clean and organise data, reducing the amount of time analysts spend on manual data processing.

2. Providing suggestions

Generative AI can provide suggestions for possible actions or directions for continued research by analysing data and identifying areas where improvements can be made. This allows analysts to generate hypotheses for further testing and exploration.

3. Creating presentations

Generative AI can assist analysts in creating professional-looking presentations without the need for specialised design skills. The new generation of AI tools currently in development can automatically generate charts, graphs, and other visual aids based on data input. AI tools can also generate text summaries of data insights, reducing the amount of time analysts spend on manual report writing. Finally, AI tools are excellent at transforming the style and format of data to suit a particular audience or purpose. This allows analysts to quickly customise their output to fit the needs of stakeholders and decision-makers.

And as a final summary, ChatGPT concluded:

Overall, the capabilities of generative AI can significantly enhance the effectiveness and efficiency, enabling analysts to generate insights and recommendations with greater speed and accuracy.

1. Insights – High impact 

Probably the most time-consuming part of our CRO work is sifting through numerous user behaviour studies, logging onto Google Analytics, checking the conversion funnel data, perhaps exporting the data, transforming the data with some Python scripts, and importing the data into R for statistical analysis. All the work of moving data, converting data, comparing data, and summarizing data have a HUGE potential to be automated with data tools.

Imagine a prompt like this:

”Watch these 10 user testing videos and these 200 session recordings. List the top 5 struggle points identified and correlate each issue with the corresponding metric in Google Analytics.”

Is this possible today? Not yet, but it might be closer than we think.

Things that definitely are possible today are to have extensive amounts of text analyzed and summarized. Ever ran a poll and got a LOT of answers? Generative AI can summarize it for you. And something that is available today with just a little bit of prompting in ChatGPT is to have analysis code produced for you. If you know your data, and have a clearly defined question, in normal natural language, you can skip navigating through web analytics interfaces and just produce code that performs the analysis and gives you a report. No prior coding skills are needed (even though that might help a little bit).

2. Ideation – Some impact

In the next step, we transform our insights into outputs in the form of hypotheses for improvements. And this is where the “human intuition and experience” comes into play. As we often say – “the data and I will tell us WHAT is happening, but it’s much harder to understand WHY it’s happening”. This type of advanced reasoning is much harder for the AI to crack. So, in this crucial step, we will have to rely on good old humans for quite some time still. 

But there is one role that the AI can play here. The AI tools are quite good at generating “seeds” for hypotheses at high speed and high volume. 

Imagine a prompt like this:

”We have seen that many users on mobile with more than 3 items in their shopping basket are less likely to complete a purchase. Give me 5 reasons why this is happening.”

When I asked this question to ChatGPT today, without providing a specific context of specific data, I received a very general answer about possible technical issues and checkout struggle points. It gave me nothing I didn’t already know. But given time, more and better data, and context one day one of those 5 answers will be something you had not thought about at all. Now the AI helped you expand your horizon and solution landscape. But in the end, the validation, the selection and the final decision on which of the hypotheses to choose, will need to rely on human expertise, in the foreseeable future.

3. Experiment – Small impact

“Small impact” – really?!

This might seem like a completely contradictory answer, but the reason is that AI and algorithms are already helping us behind the scenes on most of the AB-testing platforms and other experimentation tools. The tools do the work for us and our role is to define the settings and the guardrails around the experiments, and this is probably something we do not want to concede to the AI tools at this stage.

We’ve held workshops at Conversionista where we have used AI tools to help us with experiment design and analysis. Oftentimes the tool quickly gave us an answer identifying the winning variation and the corresponding conversion rates. Great! But: As we dug deeper into the answers, we found that the tool had picked a statistical method randomly, so sometimes the analysis was based on a Chi-square model or a T-test, a double-sided test, or a one-sided test. If you know a little about statistics, you know that these methods may yield completely different results, and selecting the correct method is crucial. A few times when the results looked pretty off, we asked the tool to take a second look and then it conceded that it had made a mistake the first time.

For tool providers, this is of course the area they should be watching most closely, but for CRO practitioners, we should simply expect providers to be in front of the AI game and have all the latest functionality included in the tools.

4. Reporting – High impact

Second, only to the data crunching, reporting is a huge time vampire in the CRO work. And in many ways, it’s the least value-adding component of all the things we do. We already have all the data, all the results, all the conclusions, and now we just need to spend hours and hours on polishing that stuff into a presentable format. Thank God, we’ll be able to trust AI to do this work for us in the near future.

Now  – what does all this mean for our practice?

We are beginning to understand the nuts and bolts of the transformation of our industry, but what does that mean for the bigger picture? Will there be a higher demand for CRO experts and services or a lower demand? Will our jobs be complemented by AI or replaced by AI?

On the upside

Many CRO practitioners I meet tell me that one of the biggest bottlenecks in the work is getting through to the decision-makers in the company, making them understand the value of the work that we do, and then ultimately prioritizing it over other things they have on their plate.

As many of the tedious and repetitive parts of our work will be automated, we will free up time for tasks that add more value and as a result, we will create more value in the same amount of time. If we do create more value it will be easier to show these to the higher-ups and get continued approval. If what we do is more valuable, companies should want more of it, right?

Another potential upside, if indeed we do become more efficient practitioners of CRO, is that websites and digital experiences will be better in general. As most of you who are deep into this field would know, there is always untapped potential left to uncover and new user insights to be revealed. So raising the bar in this competitive landscape, more user-friendly websites is not an unreasonable result.

On the downside

First off, AI can be great for crunching numbers and data, but it can miss some of the small details. It’s kind of like how you’d read between the lines of a book or a movie plot. So, even with AI in the mix, we’re still going to need human brains doing some of the thinking.

Then there’s the matter of data quality. An AI is like a chef – if you give it bad ingredients, you’re going to get a bad meal. If the data’s off, the AI could end up making decisions that are off as well. We still need humans to keep an eye on things and make sure the AIs got the right stuff to work with.

When it comes to ideas, AI can find it tricky to think out of the box. It can generate ideas based on what it already knows, but it might not come up with those game-changing ideas that a human could.

There’s also a risk that we might start relying too much on AI. It’s a bit like using a calculator for every sum and forgetting how to add it up in your head. This over-reliance could cause a skills gap in the industry over time, as fewer folks develop strong analytical and statistical skills.

Next, there’s the job situation. While we’re all hoping AI will work alongside us rather than replace us, there is a risk that some roles within the CRO process might become unnecessary. This could mean some people losing their jobs or needing to learn new skills.

Lastly, there’s the issue of understanding what the AI’s telling us. It might not always use the best method to analyze the data, which could lead to wrong conclusions. We need to double-check what the AI’s doing and not just take it at face value.

So, while AI could really shake things up in CRO, we’ve got to be smart about it. We should see AI as a tool to help us, not as something that can replace human creativity, intuition, and ethical decision-making.

I am an optimist!

Unfortunately, the whole debate around AI is skewed toward the question of how AI will replace us. The much more appropriate question is how AI will complement us. And I think there are few professions where there is so much potential for a senior role co-pilot to help us with all the boring stuff so we can do all the fun stuff. Then we can enjoy our work more and deliver higher value to our employers and clients.

Let’s be careful, considerate and make this into a dream scenario, a world without digital experiences that suck!

Want to know even more about AI and the future of CRO? Thought so.
👉🏻 Then you should definitely not miss out on Conversion Jam on October 26.