It will come as no surprise, but like most things in life, conversion optimization will benefit from a strong strategic approach.
This generally includes aligning your goals and resources to build out a roadmap, or at least a framework/process, for your experiments.
Much like in other disciplines, while experts agree that strategy is important, they can sometimes differ in their approach to such a strategy. This article will outline a couple approaches, and by the end of it, you should have a good idea of how you’d like to build out your own experimentation roadmap.
Table of contents
What’s the Importance of Maintaining a Testing Roadmap?
How do you decide what to test? It’s a question that is answered differently depending on the expert you ask. Most have some sort of discovery process, where they either conduct conversion research or at least calibrate the impact of an element with something like existence testing.
So a roadmap is a way of incorporating your prioritization framework with some sort of strategic planning of your experiments over a time period.
According to Optimizely, “A basic prioritization framework uses consistent criteria to order the experiments and campaigns you’ll run, from first to last. The more advanced version also includes a scoring rubric and an execution timeline. You’ll use your prioritization framework to manage your backlog and optimization cycles.”
So why is it vital to have an experiment roadmap, anyway?
Stephen Pavlovich, CEO of Conversion.com, outlined three advantages to building testing roadmaps:
Other experts have echoed similar thoughts in terms of how optimization is looked at from the organizational perspective (Stephen’s third point).
Similarly, Paul Rouke, CEO of PRWD, said, “building a testing roadmap underlines that conversion optimization is being taken seriously by the business. It’s time to move away from quick wins, improvisational hacks, practicing tips and tricks for increasing conversion on a whim, and taking this for what it is: a growth lever that ensures a business is sustainable and continues its growth trajectory, as well as its competitive advantage in the marketplace.”
Also important, and mentioned frequently, in building roadmaps is ensuring a holistic approach to optimization. Plan far enough out, and you ensure the capability of conducting “strategic” tests instead of small changes.
According to André Morys, CEO of Web Arts, “Most people forget that there are different strategic goals for testing. Some have the goal “growth” others want to do “research.” Some want to make sure their project works so they do “strategic” Tests. I recommend to build several testing tracks so these different main goals do not collide.”
Building a roadmap helps align teams, brand visions, and larger goals, with what you test on a week-to-week basis. It steeps your testing in the context of the broader organization and how you acheive your goals.
What’s Your Strategy in Building a Roadmap?
So you have a list of test concepts and you’ve bought into the idea that it would be smart to prioritize and plan them out over a span of time. How, then, do you do that?
There are many different ideas on this point, so I asked experts in different fields and companies – some agencies, some in-house. Here are some of their approaches…
Keep it Lean
No one can predict the future (if we could, we wouldn’t need to experiment anyway). So, we adjust our plans based on learnings. Emma Travis of PRWD put it well:
Even in terms of prioritization models, there is an inherent limitation because of the lack of prescience we have in predicting inputs. Basically, as Ronny Kohavi, Distinguished Engineer, General Manager, Analysis and Experimentation at Microsoft, told me:
A surprise win, like Ronny mentioned with the $100M Bing win, might be buried pretty deep in your roadmap. It’s hard to tell. But by keeping your process lean and reviewing your roadmap regularly, you at least open up the possibility of seeing these wins faster.
In terms of maintaining this flexibility (and incorporating learnings), y0u’ll benefit from archiving your tests and having a system wherein you can easily find and organize tests results and insights. Claire Vo, CEO of Experiment Engine, explains what that process could look like here:
I asked conversion optimization expert Andrew Anderson about his strategy in building out a testing roadmap, and he had an interesting answer:
Boiled down to its simplest form, Andrew and his team keep a huge backlog of ideas, but put them into three buckets, each of which has tests going:
- Larger tests
- Medium tests
- Small tests (“just run it” type tests)
On the topic of resources, Ronny Kohavi mentioned that “the most important thing to realize is that if something is easy to A/B test, stop the debates and just run the test.”
Or, as he put it in Controlled Online Experiments at Scale, “A key observation is that if a controlled experiment is cheap to run, then other evaluation methods rarely make sense.”
Strike a Balance
Balancing long term strategic goals with short term iterative capabilities is tricky, and there’s no real science to it (as far as I’m concerned). But it is a balance you should heed. Stephen puts it well:
Chris McCormick, Head of Optimization at PRWD, also encourages a diversity of testing strategies:
What Are the Limits or Challenges to Roadmapping?
What about maintaining it as you ramp up your testing velocity? Is it better to have a rigid plan or to iterate based on new knowledge?
No matter your strategy, maintaining a roadmap is a huge issue. It’s like Mike Tyson said, “everybody has a plan until they get hit.”
You can set a strong strategy from the start, but things will change – your resources, your insights, your results – and that warrants an approach that incorporates flexibility and iteration.
While roadmaps help you keep strategic goals in mind, committing too rigidly can also harm a testing program.
As Stephen Pavlovich put it, “iteration is crucial in testing. That’s why we recommend a test roadmap that’s long enough to deliver strategic testing efficiently, without committing to the long-term at the expense of short-term gain.”
Chris McCormick also prefers flexibility, saying, “it’s all about reacting and being agile in your approach. You may find that after the completion of one test and its analysis, you may want to follow it up with further testing based on your findings. I don’t believe you can have a test and learn culture with a rigid approach.”
It’s unrealistic to think that your roadmap is so prescient as to warrant strict rigidity. You’ve gotta incorporate learnings as you go. Paul Rouke put it really well:
What Tools Are There to Build and Maintain Roadmaps?
The answer is is pretty much the same as for any project management role. As Andrew Anderson mentioned above, you can use something like JIRA.
You can use Trello:
Or you can work from a spreadsheet. You can, of course, build your own based on your specific criteria, but Optimizely also offers one for free here.
In short, there are many effective ways of getting organized. Let your program manager figure this out and champion it to the team.
While there are many different approaches to prioritizing experiments and building a testing roadmap, we can all agree that it’s important to have a roadmap in place.
How you iterate or maintain that roadmap may depend on a variety of factors, including resources, organizational politics, or how mature your optimization program is.
A few things stuck out in terms of the expert opinions in this article: building a roadmap helps optimization be visible and gives it organizational importance, you shouldn’t be too rigid with your planning, and you should plan to maximize your resources.
Do you have a road mapping strategy in place?