Here’s something not many people talk about: no one at your organization really wants optimization to succeed – at least not in way that is most powerful and revenue impacting.
Let that sink in.
They will all talk about it, they will assist in practice, they will push you to do more and they will say all the right things. But the second a result does not go there way, they will turn on you faster than it takes you to read this overly wordy introduction.
They won’t do it maliciously, they won’t even do it consciously in most cases, but turn they will.
It is not done out of spite. They are acting in what they believe is good faith. It’s just that reality is being revealed to be different than the mental model they had been operating under, and this revelation will make you and the optimization program persona non grata. This moment, and all the subsequent ones that will happen when you run a program, will determine your success or failure. Cognitive dissonances is a powerful force, one ignored far too often in optimization.
The first step, before diving in to solve the problem, is to understand why each group will turn and what is driving that behavior. One must first understand a problem if one is to defeat it.
Your Team: The Lineup
All types of employees, whether intentionally or not, can have embedded interests that go against the success of your optimization program. Here are some of the roles and the corresponding difficulties you’ll have to face with them…
Designers
I have already covered extensively some of the misconceptions that design folks have, but I want to talk about the realities of what optimization does to their work.
Many want to see optimization as a way to validate their opinions and to see their changes go through. Some may even want to “learn” from test results that will help focus their design, but few are ready for the view shattering cognitive dissonance that can come from unexpected and truly high beta results.
I recently went through this as my team decided to run a font styling test on of our larger sites. We included concepts from our designer, our copywriter and others, but of course we included challengers to those beliefs as well to make sure our pool was as high beta as possible. So guess what won? What is the one font that would upset a designer or copywriter the most?
If you said Comic Sans, you win no prize.
Not only did it prove to generate a 70%+ lift to RPV, but it blew away all the other tested experiences. We are talking well over 3 weeks of data as proof here. For us that means we are helping 70% more people get help, which is what we are here to do. For a designer however it means that they now are being asked to put a despised and hated font on a site that they “own.”
Now I am sure that you are already going through a thousand and one excuses, explanations, attacks on testing data, etc…but that is exactly my point.
From an optimization standpoint, Comic Sans could just as easily be called “Font Variant #5,” but because we all have a visceral hatred of Comic Sans and that does not mesh with our notions of aesthetic beauty, good design, or professional pages, we must come up with an explanation to our cognitive dissonance.
Is there anything inherently wrong with comic sans? No. But from a design perspective it challenges the vision of so many. Did testing make comic sans the better option? No. It just revealed that information to us and made us face that knowledge head-on.
If you are testing in the most efficient way possible, you are going to get these results all the time. These are the moments that teams will turn against your optimization program in a heartbeat.
Brand
Continuing the example from above, you can see how the brand team might have concerns.
Obviously, they have a vision and are trying to convey it in a certain way. Most people don’t envision “Font Variant #5” as part of that plan. It takes a certain kind of ego to understand that you have less control over brand than you think you do, and that your users have far more control than you.
Does it hurt the Brand? We have zero evidence that it does. It might, but the fact that your “evidence” of that is based on some preconceived notion about a font says more about how little control we have on brand than anything the specific change can say. What we do have evidence is that more people are now going through our site and are experiencing our services, getting help that they apparently would not have gotten before, which is what the end goal of brand is supposed to be.
Development
A lot of dev teams view their site(s) as their babies, taking a protective and personal attachment for things that they have spent so much time putting together.
Clearly, something they have to manually change can cause discomfort. You can see the visceral reaction when the results are discussed and the ask comes in to make the change permanent. How can they, or any group, be taken seriously when they have to make a font change to “Font Variant #5?”
More importantly, development teams are often the most disconnected from the end result of their actions. They are used to a certain process: idea comes down, break it into components, build them, ship them, and move on. They experience very little connection to the results of those changes or to the vital cycle of improvement post initial launch.
Senior Management
Okay, all senior managers are not made the same.
But the key thing to understand is that they are the ones being rewarded for the system and the people who are in place. They have the least reason to make real changes or to challenge people’s beliefs, as that will inherently cause people to question their role in things. When something so obvious comes along and can cause grumbling, what are they to do?
Of course they want numbers, but they care far more about not rocking the ship than they care about steering it to potentially unknown places. They are, for the most part, less impacted by future numbers than they are divided in their empires that they built.
Is that what is best for the company? Rarely. Is it best for them? Yes. Are they sitting there holding a conscious weighing of the options? Rarely. Instead, they fall back to their “gut” that lead them to the current position they hold. It is always easy to reshape the facts to fit a world view if there is an obvious gain to be had by doing so.
Analytics/Existing Optimizers
Without a doubt, this can be the hardest group to get support from. Not only can they be the most dogmatic, but they also already feel like they have been working in a way that gets results.
It is one thing to work with a group that has their beliefs but doesn’t pretend to have all the answers, but it is another, entirely, to work with one that thinks they know what they are doing – when, in reality, most do not.
We are wired to believe that because we got a positive outcome, we somehow deserve a gold star. This completely ignores whether or not we could have gotten a far better outcome with the same or fewer resources.
Let’s look at the example we are working with.
- Was there data to support testing “Font Variant #5?” No. Analytics only looks at past behavior which means amongst other things it suffers from the graveyard of knowledge.
- Was there a “hypothesis” for that variant? No. And thank God, because if we only did what people thought would work, we would never have gotten this information or gotten anywhere near the results (higher performer of the favored options was 12%, so we are talking a massive difference in outcomes as well as likelihood of success).
But see, if you followed “best practices” and/or aren’t willing to see that there are many ways to tackle a problem (some may be more efficient than what you are doing), you will not only limit what you can get, but you will also enable others to limit what they can get.
That visceral reaction a great many of you experienced when you read my Comic Sans example is part of the problem. If you can’t get past that and just look at the outcome for what it is, then others who view you as knowledgeable about this subject will take that as confirmation that their cognitive dissonance is valid.
So What Can You Do?
Education
The first and most important is education.
Before I was ever hired into my current role, I discussed this exact type of situation. Then when I came onboard, I worked with my senior sponsor to make sure that they understood why and when this will happen.
I made it part of every conversation and made it a focus of the program. I view these moments as gold mines, the best moments of the program, as it means that we really proved ourselves wrong. But if you aren’t ready for moments like that, you might as well kiss your program goodbye.
I also went on a road show with the heads of each department to discuss what, why, and how we were going to act. I wasn’t looking for complete buy-in. I was, instead, focusing on creating a common language that we could use to assure that when that moment comes we would know where each party was coming from. Don’t ever expect people to like most changes, or even yourself to like the winning variant. All you can ask is that, at the end of the day, you do what is best for the business over anyone’s personal discomfort.
Bring in Groups Ahead of Time
This leads to point two, which is: bring all groups in ahead of time.
What that means is that we are constantly driving teams to invest test ideas and variants to tests that are coming. We make sure that they know this will never limit a test, but that we value all inputs.
From an optimization standpoint, the goal is simply to build the best and most efficient pool of variants. It is absolutely not to value one type of idea over another. We want them to have an input, as long as they understand that we have no control over whether it wins or how well it performs.
Vote
Finally, we encourage everyone to vote on all tests. By seeing all the variants, and constantly seeing how few are even voted for, people start to see the pattern that their opinions are not actually connected to the results.
It is not meant to attack people, but instead to bring people together to humble us together and to celebrate being wrong. It is never about us versus them, it is about all of us on a journey together.
Epilogue
For those wondering, after we had the results of this test, we went through a couple of weeks of pain.
A number of conversations were held across teams, but in the end, thanks to the best leadership that I have been lucky enough to work with and because of all the prep work we had done, we were able to move on.
Not a lot of people liked the specifics of the change, but everyone was able to join together for what was best for the business.
Since that time, we have gotten more ideas into the program, we have had more people voting on results, and of course we have taken similar tests to other sites to see if they have the same impact there. The fact that we can go so far from “best practices” and see major improvements opens eyes. I can tell you that Comic Sans is consistently one of the best performers, and that it is now on a number of our medium sites.
I can also tell you that it is no longer live on the original site, as a V2 of that test gave us a different winner with another 10% gain from changing to a somewhat similar option in Overlock.
Part of any optimization conversation is that there is no longer a static form of any site. Any change made is just a step towards a future state. Getting caught up on what won is like worrying about what ripples a rock thrown into a lake will make.
Conclusion
Cognitive dissonance is the hardest and most destructive problem an optimization program will face.
We we live in a world where an average of 90% of tests are “failures,” you are always going to be faced with an outcome that differs from what you expect. It only escalates when you attempt to maximize gains by betting on this disconnect. It takes a deep understanding and a willingness to act to turn the tide and make the most of what your organization can and should think about optimization.
In the end all the pain and effort was worth it, as we were able to use that inherent distrust and fear of optimization to grow and overcome – not to shrink or fail. With our without amazing senior leadership, it’s vital that you create these situations and proactively prepare your organization for them. The truth is, in optimization, the more often we prove our own perceptions wrong, the better the results we are getting.
That being said, if we did not first identify the these organizational challenges and take actions to deal with them, our program would be dead in the mud with little chance of pulling out. We are far from perfect but as an organization we are so much better, not just in terms of optimization but across the board, by going through the effort to create a common dialogue.
By changing it from being about the result or validation of an idea, and instead making it about the process and working together, it stopped being about them not wanting us to succeed, and more about wanting to succeed together.
Andrew, this is so good.
We don’t know what we don’t know.
And we don’t know most of the things we do know.
Test everything then do what’s right for the business.