As of next week, I, Tommy Walker, will be leaving my post as editor of CXL.
I’ve learned so much in this past year, and I owe so much to this blog. If you’ll allow, I’d like to share the 6 major lessons I’ve learned as the editor of this blog.
6. “Conversion Optimization” Isn’t About Green Or Red Buttons
People aren’t magically drawn to a button because it’s red, nor are we automatically pre-disposed to sign up just because you added the word “Free” in your headline. Optimization, isn’t about seeing which tests are “better”, but rather about studying your visitor’s behavior and creating designs & copy to engage in a dialogue with your visitor’s inner voice.
Questions like “What color converts best” are a complete waste to time. Instead, you should be always be seeking to find answers to questions such as:
Where do most people get stuck in the buying process?
What are common traits among our paying customers?
What hesitations do our leads have that prevent them from buying?
The smartest thing Peep taught me is to start closest to the money and work backwards from there.
Find out where your site is leaking money, then create a testing plan moving forward. If you aim to understand your visitors real motivations & hesitations, you’ll start running tests with substance that can turn into even more insight.
5. There Is No Failure, Only Learning
This was a huge shift in mindset for me. It’s all too easy to get emotionally involved with a test only to find that your challenger made no difference, or worse, decreased conversions.
But there are many reasons for a test not “winning” that aren’t always as cut and dry as the challenger page being worse, some of which include:
Outside influences out of your control influenced behavior unfairly
There are a large number of reasons why a test might fail on paper that are caused by inexperience & understanding what “failure” really is.
But, if I’m being honest, the real reason my early tests failed was because they were informed by my ego, not by real data.
That’s why most A/B tests fail. It was only once I started getting comfortable with the data that I stopped taking “failure” so personally. So I mis-interpreted a bit of feedback, or my colleague’s variation did better on this testing cycle… so what?
What did we learn?
Maybe “Free” wasn’t the thing they cared about. Maybe it was, but the offer still wasn’t entirely clear. Maybe an international celebrity died, and the internet had better things to do while they mourned. Maybe our visitors don’t appreciate Pre-Christmas sales starting in November.
Try not to take things so personally. Do your homework, follow the data, be smart about your tests & see what happens. If it doesn’t work, move on. There are lots of other things to test.
4. Incremental Gains Are Far More Realistic
On a similar note, try not to get discouraged if the only gains you’re seeing are small like 5%
Blog coverage of conversion rate optimization skews towards double, triple & quadruple digit gains, but that’s only because it’s sexier to cover.
Much like media coverage tends to overstate war casualties, even though we’re statistically seeing a decline, coverage on CRO looks at huge wins because it makes for much more compelling reading. But it’s far from the “norm.”
Grigoriy Kogan talks about “The Problem With A/B Test Success Stories” and how blog coverage creates an unrealistic expectation for companies who are beginning to experiment with split testing.
The trouble, he states, is that when blogs report 30-50x increases, it creates a “survivorship bias” that minimizes the impact of more realistic wins, as well as learning from neutral & failed tests. As a result, companies tend to dismiss 5% gains as insignificant & not worth paying attention to or fully implementing.
But when compounded, a 5% increase in monthly checkout completions, for example, could increase average checkouts by 60% by the end of the year.
What would a 60% increase look like for you?
3. Case Studies Are Often Big Fat Liars
At the very least, they need to be scrutinized with extreme care.
I’ve read a lot of studies that report significant lifts in conversions, but I also know from my own experience that a lift today can be neutral tomorrow.
The case study is only as good as the person managing it.
If they’ve set up their analytics improperly, or are calling wins too soon, they’re reporting “wins” that may not entirely be valid.
I’ve grown to become extremely skeptical of other people’s success stories as a result of this. With many of the case studies I’ve published here, I will research the author to see, in a broader context, they appear to really know what they’re talking about.
When you’re looking at case studies or behavioral research, there are several things you need to also take into account:
These, among many other variables, play a huge role in whether a test was actually successful. In many cases, case studies report on a false positive – the test appears to be a winner now, but in reality the “win” a temporary increase & everything returns to a neutral state a month later.
“If your only goal is to increase the conversion rate, that’s easy. You’re selling laptops for $1000? Well, sell them for $5 each and your conversions will go up 100x times at least, guaranteed. Done, goal achieved! Time for champagne?”
In the unlikely event you are able to get all of the test conditions within a case study, you have to also consider that the study is being published for PR reasons, not necessarily to provide you with insight.
That’s not to say you shouldn’t trust anyone, but rather, try not to hang your hat on any one case study or take any tactic as an absolute truth.
When you’re analyzing a case study, look at all of the information being presented to you, and really think critically about whether the lift they’re reporting is real & sustainable.
If the study is older, look to see if newer variations are building on what was found in the case study you’ve just read. There have been plenty of cases where I’ve found newer designs abandon the “learnings” found in the case study. While this could be due to internal politics, it is more likely that in the long term, the case study was actually reporting on a false positive.
As a general rule, I try to look at case studies not to understand “what” worked, but “why” it worked. If I can understand those principles, it can be incorporated into future testing.
2. Only Benchmark Off Your Own Data
What do you call the medical student who graduated at the top & bottom of their class? Doctor.
While it’s tempting to look to industry benchmarks as a way to “check in” on your performance, what you’re looking at is a warped average of the best & the worst being compiled together.
Even though, Larry Kim is speaking in the context of PPC ads, he raises a good point:
“We recently analyzed thousands of AdWords accounts with a combined $3 billion in annual spend and discovered that some advertisers are converting at rates two or three times the average. Do you want to be average, or do you want your account to perform exponentially better than others in your industry?”
That said, it’s also important when you’re creating benchmarks, you’re not looking at your own averages, but rather the segments within your traffic.
How do “Big Spenders” use the site? What about mobile users, or visitors that come from Facebook, or internal site searchers?
Instead of trying to make improvements to your overall averages, break down the performance of specific segments, and create an A/B testing plan tailored towards those segments.
It’s much easier to keep track of, and can be tackled way more systematically.
1. Stop Trying To Optimize For Stupid Crap That Doesn’t Matter
I used to think that “optimization” meant getting more people to click the button, or staying on the page longer, because those were the things that would lead to more sales. But that’s way too topical and short-sighted to be anywhere near the truth.
The answer I am always trying to arrive at now is “How does this make more money?” and it needs to start with the real data.
If I notice site searchers convert at higher rates, but the search function isn’t so noticeable, then it makes sense to try and draw more focus to site search.
If I notice that “time on page” is high for converters, by itself, that means nothing, and is likely not an accurate reflection of how long they were on the page anyways. If anything, I’d dig into the heatmaps to see what where visitors were spending the most time. I may also conduct a bit of real life user testing to identify any crappy ux issues on the page. I’d look at what pages they were on before hand, where they go after, and a whole lot of other things, in order to understand why the people on this page were converting.
But if I just tried to keep people on the page longer or more people to click the button… what’s the point if it doesn’t lead to more (and better) sales?
Conclusion
So my dear CXL reader, those have been the 6 most important lessons I’ve learned on my journey as an editor here. I encourage you to take a look at everything I’ve linked to, and even if you’ve already read those pieces, read them again.
We’ve worked very hard to put together some of the most comprehensive information in Conversion rate optimization online, and hope that you’ve been able to profit from it.
As for me, I’m moving over to Shopify to continue writing about CRO specifically for eCommerce. I love CXL, it’s readers & the connections I’ve made here. You are such a valuable part of my life now, and I appreciate the time you’ve spent reading, commenting on, and sharing my work here.
Don’t worry though, CXL isn’t going anywhere; Peep and the rest of the team have some really great stuff in the works that you can’t afford to miss.
