So, the revenue in your variant is actually lower – even though the conversion rate is higher!
While this may seem as a simplified example, it puts the spotlight on some of the main issues:
- The presence of outliers, and how they’re distributed across variants, may severely impact your inference for the revenue. Had the outlier in this case been present in the variant, we would instead have grossly overestimated the revenue effect of our treatment
- How to handle such outliers is by no means a trivial matter; there’s no one-size-fits-all answer
- In general, revenue data will be a lot more noisy than the simple convert/didn’t convert figures
- The order value can, in its own right, be considered a random variable. Depending on the circumstances, it may or may not be influenced by the experiment setup. The statistical test you performed, however, was not for the order values but for the conversion rate. The uncertainty in the revenue estimate is left unaccounted for
- It should be noted that the conversion rates in the variants are also estimated, with their own levels of uncertainty.
… and that’s not all
Apart from what’s mentioned above, there are several other factors of uncertainty as well. For example, the number of conversions will of course depend on the traffic to your page. This is yet another random factor that has not been accounted for. External factors like seasonality, economy, competition and so on may influence both the conversion rate and order values in the future. Such effects can be both negative and positive.
Furthermore, the effect of the variant you implement may wean off over time or even be limited to a specific time period. It should also be noted that, when dealing with noisy data like revenue and doing experiments with low power, your estimates may be subject to what’s called Type M error. That is, you’ve overestimated the effect.
Last but not least, the standard methods used to analyze experiments are statistical hypothesis tests, not forecasting models. Once you start predicting future effects, you are taking a leap into uncharted territory.
That being said, it’s not unreasonable to assume that a variant which increases the conversion rate will have a positive effect on revenue. But to put an exact measure on it, and to predict future revenue, is evidently a lot more complicated.
So what do you tell the management?
It’s not necessarily wrong to make some naive calculations – like multiplying the revenue effect by 12 – and presenting it. If you find that this is the most straightforward and easy way to do it in your case, by all means, do it.
However, I believe it is imperative to be very clear and transparent about which assumptions you’ve made, and the uncertainty in your measures. Here are some suggestions:
- Include confidence and/or credible intervals of your estimates
- State explicitly which assumptions you’ve made (“all other things equal”)
- Always check for outliers, and give an account for how you’ve handled them
We know it can be a bit tricky sometimes, but remember – we’ve got your back! Swing by our local office for a coffee, give us a call, or send an email to hej@conversionista.se and we'll happily deep dive into this together.
More reading:
Can you really tie A/B testing to revenue? – Collin Tate Crowell, Kameleoon.com
Bayesian A/B Testing at VWO (includes methods for modeling revenue) – Chris Stucchio, VWO
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors – Andrew Gelman and John Carlin
An Introduction to Type S and M errors in Hypothesis Testing – Andrew Timm
Overcoming the winner’s curse: Leveraging Bayesian inference to improve estimates of the impact of features launched via A/B tests – Ryan Kessler, Amazon
