It will come as no surprise, but like most things in life, conversion optimization will benefit from a strong strategic approach.
This generally includes aligning your goals and resources to build out a roadmap, or at least a framework/process, for your experiments.
Here’s something not many people talk about: no one at your organization really wants optimization to succeed – at least not in way that is most powerful and revenue impacting.
Let that sink in.
Just when you start to think that A/B testing is fairly straightforward, you run into a new strategic controversy.
This one is polarizing: how many variations should you test against the control?
The traditional (and most used) approach to analyzing A/B tests is to use a so-called t-test, which is a method used in frequentist statistics.
While this method is scientifically valid, it has a major drawback: if you only implement significant results, you will leave a lot of money on the table.
Right now there is almost certainly an enterprise exec in a boardroom somewhere saying, “We need it to go viral.” Kittens and memes and babies kissing puppies… viral.
When most people think about going viral, they think about raising a lot of awareness for their product or company. But what about money in the bank, what does going viral mean for your bottom line?
So, the statement becomes: We need to go viral in a way that makes us actual money. Not surprisingly, that usually looks a little different than kittens and memes and babies kissing puppies.
If you’re doing it right, you probably have a large list of A/B testing ideas in your pipeline. Some good ones (data-backed or result of a careful analysis), some mediocre ideas, some that you don’t know how to evaluate.
We can’t test everything at once, and we all have a limited amount of traffic.
You should have a way to prioritize all these ideas in a way that gets you to test the highest potential ideas first. And the stupid stuff should never get tested to begin with.
How do we do that?
A/B testing is common practice and it can be a powerful optimization strategy when it’s used properly. We’ve written on it extensively. Plus, the Internet is full of “How We Increased Conversions by 1,000% with 1 Simple Change” style articles.
Unfortunately, there are experimentation flaws associated with A/B testing as well. Understanding those flaws and their implications is key to designing better, smarter A/B test variations.
As an optimizer, you might be thinking that user interviews fall outside your role. Or, perhaps, that they are a “nice to have” on the qualitative conversion research checklist. Worse, you might not be asking good survey questions because you’re rolling with an “I’ll just wing it” mindset.
User interviews are more complex and important than most optimizers realize.
As marketers, we’re all trying to improve the customer experience and increase conversions. We have these things in common.
However, some marketers are much better at understanding their customer personas and doing the right kind of research than others.
What is comes down to is that delivering a single message to your entire customer base is an inherently flawed strategy. High-value customers, frequent browsers, seldom purchasers, brand enthusiasts and first-time visitors are all differently characterized and must be engaged uniquely.
This is where customer micro-segmentation comes into play.
I have been part of some the best conversion optimization teams in the world, and they seem to have an intuitive sense on how to run the best experiments. People that are involved in these teams share a similar mindset.
I wanted to try to make this a process, one that could teach any organization how to run better experiments. I wanted to try to make this mindset more explicit in a way that is fun to use.
There’s no rocket science that follows, but this framework may well help your team drive a more efficient optimization culture.