Turning the tide: how deep analysis at Fortum boosted A/B test results from -9% to +9%

About Fortum

Fortum is a Nordic energy company with a mission to power a world where people, businesses, and nature thrive together. As one of the cleanest energy producers in Europe, Fortum's actions are guided by ambitious environmental targets. The company generates and delivers clean energy reliably and helps industries decarbonize their processes and grow. Core operations in the Nordics include efficient, CO2-free power generation and the reliable supply of electricity and district heat to private and business customers.

Challenge

When Conversionista partnered with Fortum, we identified significant opportunities to enhance their data-driven approach. The team lacked a clear understanding of the user journey, pain points, and optimization priorities. Despite having numerous ideas, there was no process to measure the impact of these ideas. This is where Conversionista stepped in.

Solution

Our goal was to establish a data-driven culture with analytics at its core. We helped Fortum build a process to convert ideas into testable hypotheses using data triangulation. We prioritized A/B testing due to limited traffic and resources, ensuring thorough post-test analysis for maximum insights.

The Test

We collaborated with Fortum to run an A/B test on the price page within the purchase flow. The objective was to redesign the pricing blocks to clarify discounts and streamline information delivery. The primary KPI was the click-through rate, with additional tracking of user interactions with price details.

Initial Result

The first A/B test resulted in a -9% click-through rate, indicating a need for deeper analysis.

Deep-Dive Analysis Process

Price Details Interaction:

Interaction Levels:

Click-Through Rate Post-Interaction:

• • Noted a drop in users progressing in the funnel after viewing the second level of price details in the variant (18% vs. 75% in the original).
  • Suggested the modal interaction was problematic.

Conclusion and Learnings

The analysis suggested that while the variant made price details easier to find, the modal interaction hindered user progression. We redesigned the variant to retain the drop-down function, enhancing clarity with improved copy and color usage. The subsequent test with these changes yielded a +9% increase in the click-through rate.

Results

• +9% Click-Through Rate: The redesigned variant significantly outperformed the original.
• Enhanced User Insights: Deep analysis provided valuable insights into user behavior.
• Improved Experimentation Process: Highlighted the critical role of thorough analysis in optimizing A/B testing outcomes.

Key Takeaway

Deep analysis is essential for extracting maximum value from A/B testing, guiding informed decision-making and driving continuous improvement in user experience and digital performance.