A strong A/B testing plan will allow you to increase your revenue. You’ll also learn valuable insights about your customers because you’ll know their preferences instead of guessing what they want.
A/B testing produces concrete evidence of what actually works in your marketing. Continuously testing your hypotheses will not only increase conversion rates, but it will also give you a better understanding of your customers. Having a clear idea of what your customers actually like and prefer can do wonders for your branding and marketing in other channels as well.
At its core, A/B testing belongs to a category of scientific optimization techniques that use statistics to increase the odds that your site visitors will see the best-performing version.
By levels of sophistication, scientific optimization can be broken down into three categories:
- A/B split testing. Simple testing of one element of a page against another to see which one results in better performance.
- Multivariate Testing. Testing several elements at a time. The goal is to get an idea of which elements work together on a page and play the biggest role in achieving the objective.
- Experimental Design. Developing your own research method for an in-depth analysis of a specific element.
This post is about the A/B testing process. A good A/B testing plan produces the fastest gains and has a lower chance of error through misuse.
A structured A/B split testing process
A/B testing is a part of a wider, holistic conversion optimization process and should always be viewed as such. Doing A/B testing without thinking about your online goals and user behavior in general will lead to ineffective testing. When correctly planned, you can achieve good, measurable results in a short timeframe.
A structured process to improve your conversion rates needs to be continuous. It is a cycle of Measurement, Prioritization, and Testing (then repeat). Each stage has a goal and a purpose, leading to the next stage.
For your plan to be successful, follow this four-step A/B testing template.
Step 1: Measure your website’s performance.
To continually improve your conversion rates, start by properly measuring your website’s performance. We want to know what is happening and why it’s happening.
What is happening: Get actionable data from Google Analytics
Define your business objectives.
This is the answer to the question: “Why does your website exist?” Make your objectives DUMB—Doable, Understandable, Manageable, Beneficial. Companies often fail in web analytics because their objectives are not simple to understand or measure.
Example: A business objective for an online flower store is to “increase sales by receiving online orders for our bouquets.”
Define your website goals.
Goals come from your business objectives and are mostly strategic in nature. So, if we were to continue with the business objective example of increasing our bouquet sales, we have to:
- Do x. Add better product images;
- Increase y. Increase click-through rates;
- Reduce z. Reduce our shopping cart abandonment rate.
Goals are your priorities, expressed as simply as possible. Before you start working with your data, make sure you have them defined and properly set up in Google Analytics.
Define your Key Performance Indicators (KPIs).
KPIs are metrics (numbers).
“A key performance indicator (KPI) is a metric that helps you understand how you are doing against your objectives.” – Avinash Kaushik
A metric becomes a KPI only when it measures something connected to your objectives.
Example: Our flower store’s business objective is to sell bouquets. Our KPI could be the number of bouquets sold online.
This is the reason why you need to define your business objectives clearly—without them, you’re unable to identify your KPIs. If you have proper KPIs and look at them periodically, you’ll keep your strategy on track.
Define your target metrics.
Our flower store sold 57 bouquets last month. Is that good? Or devastatingly bad? For your KPIs to mean something, they need target metrics. Define a target for every KPI. For our imaginary flower store, we can define a monthly target of 175 bouquets sold.
Now you have a framework that ensures that the work you do is relevant to your business goals.
Why it’s happening: Talk to your visitors
Getting real feedback from your visitors is invaluable. Use surveys to discover your visitors’ objectives. Set up entry surveys to find out why they’re visiting your site and exit surveys to find out if their goals were fulfilled.
You can find what is happening on your site by using Google Analytics—which features they use, where they exit, and who is profitable. But Analytics doesn’t tell you the why behind this. This is where qualitative data comes in. Qualitative data is perfect for finding out why problems occur.
The best way to get qualitative data from your visitors is through surveys. Your goal is to find out why visitors buy or why they leave without purchasing.
Ideas for gathering qualitative data
- Add an exit survey on your site, asking why your visitors didn’t complete the goal of the site.
- Add an exit survey on your thank-you pages to find out why your visitors converted.
- Perform usability testing with members of your target group.
- Send out feedback surveys to your clients to find out more about them and their motives.
If a large amount of people click on your ebook ad but only a few people actually buy after seeing the price, you’ll want to dig deeper into the problem.
In this example, you could put up a survey, asking people if they have any questions that the page doesn’t answer. You can also survey people who have already bought your book to see what made them buy.
Look for trends in your customer feedback
You’ll start noticing trends after you’ve collected 50+ responses. Often, you’ll find that the site hasn’t addressed an important objection of your client. The main takeaway is that qualitative data will help you understand which elements will have the highest impact when running an A/B test.
Think about how you could spot emerging trends with your customer feedback. When something pops up, you can dive deeper to find out exactly what’s going on.
Use segmentation to get actionable data
The problem with using site averages in your testing is that you’re missing out on what’s going on inside the average (the segments).
An experiment that seemed to be performing poorly might actually have been successful for a certain segment. For example, our experiment may have shown that a variation of a mobile landing page isn’t performing well. When looking into the segments though, you may see that it’s performing exceptionally well for Android users but badly for iPhone users. When not looking at segments, you can miss this detail.
Never report a metric (even the most hyped or your favorite KPI) without segmenting it a few levels deep. That is the only way to get deep insights into what that metric is really hiding or to see what valuable information you can use.”Avinash Kaushik
To understand segments, we need to understand dimensions. A dimension is any attribute of a visitor to your website. A dimension can be a source where they came from (a country, URL, etc.), technical information like their browser, or their activity on the site (pages they looked at, images they opened). A segment is made up of a group of rows from one or more dimensions.
By default, a lot of the data you get is useless. The number of visits to your site doesn’t really give you actionable information. To get actionable data for testing, you need to segment the data you have, using dimensions. You can also split test for single segments of your traffic, which is a part of behavioral targeting.
Three good segmentation strategies for your testing plan
Avinash Kaushik has outlined a simple strategy for segmenting your traffic. The best ideas for taking action come from segmentation.
Segment by source. Separate people who arrive on your website from email campaigns, Google, Twitter, YouTube, etc. Find answers to questions like:
- Is there a difference between bounce rates for those segments?
- Is there a difference in visitor loyalty between those who came from YouTube versus those who came from Twitter?
- What products do people who come from YouTube care about more than people who come from Google?
Segment by behavior. Different groups of people behave differently on a website because they have different needs and goals. For example, an ecommerce store can separate people who visit more than 10 times a month from those who visit only twice. Do these people look for products in different price ranges? Are they from different regions?
Segment by outcome. Separate people by the products they purchase, by order size, by people who have signed up, etc. Focus on groups of people who have delivered similar outcomes and ask questions like the ones above.
Keep your most profitable segments in mind when building your split-testing plan. It’s good to know who your most profitable visitors are before you start split testing.
Step 2: Prioritize your testing opportunities.
Once you have your metrics in place and know your goals, the next step is to prioritize what to test. You could test anything, but everyone needs a place to start from. Google Analytics gives you a lot of data, but it makes sense to start by split testing opportunities that promise the biggest gains.
Prioritize tests based on data—it’s your most valuable resource.
Your homepage may not be the most important area of your site. Look at your “top landing pages” report from Google Analytics. You’ll likely see many different pages with entrances, some even more than your homepage.
Look at data on a page template level.When you add together the traffic from all of your pages that use the same template, you may see that they get a lot more traffic than your homepage. Do this when you need to determine opportunities for testing site-wide template layouts.
Prioritize pages with high potential for improvement.
Look for pages that aren’t performing well. Your Analytics data can show some clearly problematic pages, like landing pages with high bounce rates, but other areas may not be so obvious.
If your problem is a high shopping-cart abandonment rate in the checkout process, Google Analytics won’t tell you that visitors can’t find shipping information on other pages and, as a result, are going into the checkout process just to see that information.
If you optimize only shopping cart views, you may not fix the problem. You also need to look at your product and category pages.
No single information source will perfectly identify split-testing opportunities. You need to look at several pages together.
Top exit pages. This is the last page that someone sees before leaving your site. Labeled “% Exit” in Google Analytics, it will show you the percentage of visitors who leave your site immediately after viewing the page. Top exit-rate pages can identify problem areas within your user flow.
You can visualize the user flow with a conversion funnel:
- Persuasive end (top of the funnel). The persuasive end includes the most-viewed areas of you site like your homepage, category pages, and product pages. These are the areas of your site where you’re getting the visitor interested in your product or service.
- Transactional end (bottom of the funnel). The bottom end of your funnel is where a conversion happens—visitors buy the product, sign up, or contact you. Most of the data we’ve looked at so far has been focused on the persuasive end of the funnel, but we also need to look at the bottom of the funnel.
Look at funnel drop-off rates.
A funnel in Google Analytics focuses on the bottom end of the funnel. If you have your funnels correctly set up, you can gain valuable split-testing information from it.
Look for sudden drop-off rates in the funnel. For example, if only 18% of the traffic proceeds from Step 2 to Step 3 in the checkout area, you have a problem in Step 2.
If you’ve identified a drop-off step in your transactional funnel, you should ask yourself why the problem is happening:
- What information were they looking for?
- Is anything stopping them from taking action on the page?
- What were they expecting to see on the page?
- Where are visitors coming from?
- Are they not motivated enough to proceed?
Answers to these questions should give you ideas for how to plan your split tests.
Prioritize tests based on value and cost.
Start with high-value, low-cost testing ideas. An example is testing variations in a checkout process step that shows significant abandoment rates compared to previous steps.
Prioritize pages that are important.
Pages that have the highest traffic volume are the most important for testing. You have probably identified many pages that perform worse than you would like them to, but if they don’t have a high volume of (expensive) traffic, don’t count them as priorities.
Pages with a high volume of traffic are more important.
You need pages with high traffic for completing your experiments within a reasonable timeframe. Pages with more than 30,000 monthly, unique visitors can reach statistical significance in a few weeks.
With a lower level of traffic, you need to run tests over a longer period of time. Because tests on high-traffic pages finish sooner, you can move on to the next tests faster, which will speed up your optimization process.
- Most-visited pages. Look for information on unique visitors. When looking at the number of overall pageviews, your conversion data will get skewed.
- Top landing pages. When looking at the most-visited pages, you will see the most popular landing pages on your site. You also need to look at the most-visited pages that people see when they first arrive on your site.
- Pages with expensive visits. When choosing between two pages with similar traffic and conversion rates, pick the one with a higher cost of traffic for a better split-testing ROI.
Use the PXL prioritization framework
CXL created their own prioritization model that weeds out as much subjectivity as possible. It’s predicated on the necessity of bringing data to the table. It’s called PXL and looks like this:
Grab your own copy of this spreadsheet template here. Just click File > Make a Copy to have your own customizable spreadsheet.
Instead of guessing what the impact might be, this framework asks you a set of questions about it:
- Is the change above the fold? Changes above the fold are noticed by more people, thus increasing the likelihood of the test having an impact.
- Is the change noticeable in under 5 seconds? Show a group of people the control and then a variation(s). Can they tell the difference after seeing it for 5 seconds? If not, it’s likely to have less impact.
- Does it add or remove anything? Bigger changes like removing distractions or adding key information tend to have more impact.
- Does the test run on high-traffic pages? Relative improvement on a high-traffic page results in more absolute dollars.
Many of the variables specifically require you to bring data to the table to prioritize your hypotheses.
- Is it addressing an issue discovered via user testing?
- Is it addressing an issue discovered via qualitative feedback (surveys, polls, interviews)?
- Is the hypothesis supported by mouse tracking, heatmaps, or eye tracking?
- Is it addressing insights found via digital analytics?
Having weekly team discussions on tests with these four questions will quickly make people stop relying on opinions.
There are also bounds based on the estimated time to implement (“ease of implementation”). Ideally, you’d have a test developer as part of prioritization discussions.
This is a binary scale. You have to choose one or the other. So, for most variables (unless otherwise noted), you choose either a 0 or a 1.
Certain variables are also weighted because of their importance—how noticeable the change is, if something is added/removed, ease of implementation.
So, on these variables, we specifically say how things change. For instance, on the “Noticeability of the Change” variable, you either mark it a 2 or a 0.
Step 3: Test.
The reason for the in-depth analysis techniques described above is to get a clear understanding about what is happening on your website. That process enables you to find areas where A/B testing can bring the best results and to plan accordingly.
Now let’s set up your test.
Form a clear hypothesis for your test.
A hypothesis defines why you believe a problem occurs. If your problem is high abandonment rate in your checkout process, your hypothesis may be that “People start questioning our worth after seeing grammar mistakes.”
After you define your problem and articulate your hypothesis, you can come up with specific split-testing variations. Clearly defining your hypothesis will help you create test variations that give meaningful results.
An example of a hypothesis in action:
- Problem. Less than 1% of visitors sign up for our newsletter.
- Hypothesis. Visitors don’t see the value in signing up for our newsletter. Adding three bullet points about the benefits will increase sign-up rates.
In this case, we would summarize the benefits that the newsletter member would get from joining the newsletter. Even if the original version works better in your A/B test, you learned something about your visitors. You clearly defined why you did the test and can draw conclusions based on the outcome.
A good hypothesis…
- Is testable. Your hypothesis is measurable, so that it can be used in testing.
- Has a goal of solving conversion problems. Split testing is done to solve specific conversion problems.
- Gains market insights. Besides increasing your conversion rates, split testing will give you information about your clients.
Use valid statistical methods.
Ignoring statistical significance when running an A/B test is worse than not running a test at all. The results will give you confidence that you know what works for your site when, in reality, you don’t know more than before running the test.
If we were to toss a coin 1,000 times, we would reduce the influence of chance but still get slightly different results with each trial.
The statistical significance of your experiment should be over 95%. But statistical significance alone is not a guarantee. It is a measure of confidence.
Test for revenue.
At the end of the day, revenue is what matters. An increase in conversion rates may sometimes mean a decrease in revenue if you’re tracking the wrong indicators.
Let’s imagine that you’re selling watches online and suddenly increased your prices by 25%. Your conversion rates will probably drop, but your overall revenue may increase if the demand for your product is high enough.
What to test—the low-hanging fruit
You should always use analytics and customer feedback to plan your tests. But for a general guideline, here are elements that historically have given good results:
- Call to actions. Placement, wording, size;
- Copywriting. Value propositions, product descriptions;
- Forms. Length, field types, text;
- Layout. Homepage, content pages, landing pages;
- Product pricing. Try testing for revenue by testing your prices;
- Images. Placement, content, and size;
- Amount of content on the page. Short versus long.
Step 4: Learn from your results and start over.
Never stop testing, and your advertising will never stop improving.
This is the one rule of marketing that surpasses all others. No matter how well your landing pages or emails may be doing, they can always be doing better. By not having a goal for constant improvement, you’re leaving money on the table and letting your competition get ahead of you.
Not all split tests will be successful. You are doing well if 20% of split tests improve your conversion rates. Split-testing will simply give you a new baseline to improve upon. You will start seeing considerable website performance improvements after a number of different split tests.
Considering how competitive today’s online market is, if you aren’t constantly tracking and optimizing, you’re going to get left behind—and outsold—by people who are.
A/B testing plan considerations
Be aware of the local maximum.
Most A/B testing is done one variable at a time. You test headlines, button text, images etc. These variables are simple to test; your results will be clear; and your next steps will be obvious. By isolating one variable, you can be more confident in your results. There’s a downside, though.
The argument against testing single variables is that, if you continue to do it for a long period of time, it will be impossible for you to arrive at a much better design.
Instead, you’ll be improving your existing design in small increments and get stuck at the local maximum. You’ll hit a glass ceiling in your design and, without a big change, be unable to earn larger gains.
Google openly endorses split testing because the goal is to improve a website and make users happy. But there are a few things you should consider.
Don’t run your experiments longer than necessary. If your experiment has been running longer than what Google would normally expect from a split test, you may confuse the search engine about which version of the content is the “real” version.
The general rule is to update your site with the winning variation as soon as you are sure of its statistical validity. Google wants to prevent people from deceiving search engines.
Use rel=canonical. Instead of using a noindex meta tag on your variation page, use rel=canonical. If you are testing two variations of your homepage, you don’t want search engines to deindex both.
You need search engines to understand that your variations are what they are—variations of the original URL. Using noindex in a situation like this may create problems later.
Use 302 instead of 301. When you need to redirect a visitor to one of you variations, use a 302 temporary redirection instead of the permanent 301 version. You want Google to keep the original URL in its index.
Forget about A/A testing
A/A testing validates your test setup—if your variations display correctly and your software reports the right numbers. The biggest problem with A/A testing is that setting it up and running it takes time (and traffic) that could be used for more split testing.
The volume of tests you start is important, but even more so is how many you finish every month.
It’s quicker for you to test your experiments before going live. To make sure your variations are displaying as they should in different browsers, use cross-browser testing. For checking the numbers, use your split-testing and analytics packages together on every test.
If, for whatever reason, you still need to do an A/A test, consider an A/B/A test, or 25/50/25 split, instead. There’s an even better way: segmentation.
Avoid the percentage confusion.
Besides significance, the other common problem A/B tests have is quoting percentages. Since conversions are measured in percentages, there are two ways to report them:
- Change as the numerical difference between the two variations;
- Change as the amount by which one variation is larger than the other.
Normally, Version 2 is used when reporting conversion rate improvements, but make sure you know this before deciding anything based on the numbers.
Integrate your A/B testing process with conversion optimization methodologies.
Your A/B testing efforts will bring better returns when you execute them within a conversion optimization framework. Different agencies have come up with different versions for this. Here are some of them:
Conversion Rate Experts have developed a systematic process for guiding a business through a series of steps toward better conversion rates.
MEClabs has a funnel that represents a series of decisions taken by the prospective customer. The idea revolves around the customer taking small decisions step by step, “micro-yeses,” which lead to the Ultimate Yes, or the sale.
The idea is to prioritize conversion testing opportunities and implement the right experiments, which will drive the most impact-filled results. This will help you maximize your conversion rate improvement.
CXL has developed their own framework that’s used by dozens of agencies. It includes six parts of research and also has a prioritization component, solving the oh-so-common problem of what to test.
The framework consists of the following:
- Heuristic Analysis;
- Technical Analysis;
- Web Analytics Analysis;
- Mouse Tracking Analysis;
- Qualitative Surveys;
- User Testing.
They’ve also written a conversion optimization guide that goes into further detail and explains how ResearchXL ties into a holistic optimization program.
Visitors interact with their mobile devices through tapping, so their ability to quickly locate and hit these tapping areas is key to mobile conversion rates.
When testing on mobile screens, begin testing elements related to the user experience and design of the site—navigational elements or mobile forms.
Mobile analytics generally fall into two categories: tracking how people interact with websites in a mobile browser and in-app analytics. Let’s briefly touch on both.
Testing in a mobile browser
A smaller screen forces users to focus more on what is relevant, giving you ample opportunities in the conversion area.
What to test for mobile users:
- Call-to-action buttons. On a desktop screen, the CTA button won’t take up more than 2% of the screen real estate. On a mobile device, you have the opportunity to make your CTA button take 25–50% of the screen real estate. You can test any standard property on your button size, copy, and visual design.
- Navigation. You need to change the site navigation for mobile viewers. This is a good opportunity to direct visitors to the most important page content. Think of mobile navigation as a tool for helping your visitors achieve their goal, and remove anything that can get in the way.
- Copywriting. Since the screen real estate of mobile devices is small, focus your visitor’s full attention on your value proposition. Test the wording carefully.
- Forms. Experiment with the design and number of fields, dropdowns, and error messaging.
Google Analytics has a mobile section, but it includes both mobile phones and tablets. Accessing a website on a tablet is a different experience from a mobile phone, so make sure you use extra filters for cleaner data.
Compare your site performance (visit duration, bounce rate, conversion rate, etc.) for mobile visitors vs. desktop visitors for insights into areas that may cause trouble for your visitors.
Use a previously set advanced segment (where you’ve filtered out non-mobile traffic) and apply that to your All Pages report. You’ll get a good overview of which pages your mobile visitors use, which may be vastly different than those viewed most often by desktop users.
This information can be helpful to test mobile menus. You can bring out the content your mobile visitors want to see the most, creating a different navigation menu from your desktop version.
For more information about Google Analytics for mobile platforms, see this thorough post by Bridget Randolph.
2. Testing in applications
Google Analytics has a separate tool for measuring and testing variations in mobile applications. The main difference from testing in a browser is that, without putting in some thought, you’ll have a slower reiteration cycle.
You need to upload every test to the app store and wait for your users to update their apps. This means you should carefully think through what you want to track and test in your application before you release it. It will be more difficult to change this after your app has been downloaded.
How is multivariate testing different?
When you perform a multivariate test, you’re not testing a different version of a web page like you are with an A/B test. You’re performing a far more subtle test of the elements inside one web page.
A/B testing usually involves fewer combinations with more extreme changes. Multivariate tests have a large number of variations that usually have subtle differences.
A/B testing is usually a better choice if you need meaningful results fast. Because the changes between pages are starker, it’s easier to tell which page is more effective. A/B testing is also better if you don’t have a lot of traffic to your site. Because multiple variables are tested together, multivariate testing needs a site with a lot traffic.
The goal of multivariate testing is to let you know which elements on your site play the biggest role in achieving your objectives. Multivariate testing is more complicated and is better suited for advanced testers. It’s more prone to reporting errors and common to have more than 50 combinations.
Case studies for inspiration
Citycliq increased their conversion rates by 90% after experimenting with Value Propositions
Citycliq was testing their value propositions to see what converts the best. Eventually, they concluded that the value proposition with the “Purest, most direct representation of their product” was the winner.
Call to action:
During the election period, the Obama team ran several A/B tests on the campaign’s landing page. The goal was to get people to sign up with their email addresses. A/B testing generated an additional 2.9 million email addresses, which translated to an extra $60 million in donations.
The challenger page increased the size of the contact form. They also replaced a general logistics image with the image of an actual DHL emplyee.
A/B testing resources
Instead of listing all of the various testing tools out there (and by now, there are many), take a look at our gigantic list of CRO tools. Each tool is reviewed by an actual practitioner, so you can choose which one is right for you.
A/B Split Test Significance Calculator by VWO
A widely used tool for calculating the significance of your A/B testing results.
A/B Split and Multivariate Test Duration Calculator by VWO
The calculator allows you to calculate the maximum duration for which your test should run.
This is for surveying visitors that are currently surfing your site. We use this in our agency. Allows you to set up behavior-triggered surveys to find out why your customers are behaving the way they do.
SurveyMonkey has been around for a long time and offers a stable experience. It has good analytical tools for analyzing your responses.
Free and easy to use. Data is collected into spreadsheets which makes the data easy to analyze.
The prettiest survey tool of all.
You should always integrate A/B testing into a larger conversion optimization framework for good results. In the end, the testing is all about knowing if your hypotheses are right and if your conversion plan is on the right course.
The main reason for doing split testing is to maximize the conversion potential of your website. It makes sense to invest in conversion rate optimization before spending money on large-scale advertising campaigns.
After a while, your conversion funnel will be effective enough that you can transfer your winning campaigns to other media. Because your core is optimized so well, by the time you go to offline media, your campaigns will convert well enough to pay off.
Key takeaways to create a solid A/B testing plan
- Make sure you get actionable data from Google Analytics. To make use of data, you need to have goals first. Define your target metrics.
- Use qualitative surveys. Besides knowing what is going on, you also need to know why.
- Segment. It’s the only way to get valuable information.
- Prioritize. Test pages with higher potential first.
- Form a hypothesis. A/B testing has to start with a clearly defined hypothesis.
- Statistical relevance. Never stop split testing before reaching a significance percentage of at least 95%.
- Never stop testing. No matter how well your landing pages or emails may be doing, they can always do better.
- A/B testing is part of a conversion rate optimization framework. Pick a framework and work on it constantly to maximize your site’s potential.