“I have too little traffic to test; hence, I can’t do conversion optimization.“
I’ve heard this a lot. Bullshit, I say. You can do conversion optimization on any website – even on sites that have so little traffic that a split test would take 2 years to run until statistical significance is reached.
You might not be able to run tests, but you can still optimize. It’s not entirely scientific, but hey, most of us are in the business of making money, not in the business of science.
Table of contents
The problem with very little traffic:
Problem Number 1
Testing on sites with little traffic will take forever to run tests until statistical significance is obtained, and time is money. If you run tests, you better have a strong testing plan and run them until there is statistical significance. Otherwise you run the risk of shooting yourself in the foot by operating with completely false data. No point in testing if the data is wrong.
That being said, remember that most optimizers run tests until 95% significance. That means there’s is a 5% margin of error. Most optimizers will take that risk.
Now, if you run a treatment that brings considerable positive change (e.g. +50%), you can also end testing before it reaches 95% confidence, like at 80% (thus saving 4 more months of waiting for conclusiveness). Is it scientific? Nope, but it will potentially make you more money.
How long will your test take? Check this handy tool for a good ballpark figure. If it’s 6 or 12 months, you’re better off not running split tests for now and optimize the site without running tests (until traffic picks up).
Problem number 2
Testing on sites with little traffic will also take a long time to collect statistically, valid sample sizes of data in your Google Analytics (and other tools, like mouse tracking stuff). Yeah, it sucks, but you can still gather data and analyze the site to come up with hypotheses for treatments.
Analyzing your website when you have low traffic
No traffic, no problem.
Heuristic analysis ftw
Start with a heuristic analysis. Heuristic analysis is an expert based analysis that uses experience-based techniques for problem solving, learning, and discovery. Its results are not guaranteed to be optimal.
While we have to be very data-driven in this job, there’s no substitution for human-led evaluation of a website. If you’re an experienced optimizer and look at a website, you will start seeing stuff right away that’s not quite right. Of course, like every human being, you can be dead wrong, and you have to look at the data to back up your initial hypotheses.
If you haven’t analyzed hundreds of sites, that’s not a problem. There are several frameworks for heuristic analysis that I’ve covered in this post – they structure and focus the process. Gather your team (include some customers and experts, if you can), and tear your website apart.
Most of the heuristic frameworks deal with the following criteria for each page on your site:
- Clarity – Can I understand the value proposition? Is it clear what the site and offer are about? Is it obvious what my next step should be? Is there one primary call to action? Is everything important above the fold?
- Friction – Are all the forms as short as possible, all processes as simple as possible? Are security and privacy concerns addressed? Main questions answered? Main doubts eliminated? Ample proof provided?
- Distraction – Are there elements on the page that are not directly contributing to people taking most desired action? Anything blinking or moving automatically? Remove all of them.
- Urgency – Is there anything that compels people to take action right away?
- Traffic sources and scent – Do people who land on this page come with the right expectations? Does the page content match what they seek?
- Buying phases – Are they researching or ready to buy? Are we asking for too much too soon?
Longer explanations for each in this post.
Ways to get useful data without traffic or many customers
There are tools, yo!
Your traffic doesn’t matter for user testing. How to do this? Read this article I’ve written to get the full scoop.
Find 5 to 10 people who represent your target audience and are not familiar with your site. Have them complete tasks on it and pay attention to these three critical areas:
- Learn how they understand your site and what its a about.
- See if they experience any confusion or difficulty when checking out/completing any business critical tasks.
- Understand how they would like to buy and what matters to them.
If you have a mass-market site, turn to sites like Usertesting.com, TryMyUI, or YouEye. If your audience is more niche, go to relevant forums (find them via Google) and post an ad that you can send them $25 Amazon gift cards in exchange for 20 minutes of their time testing your site.
Mouse tracking data can be very useful – attention heat maps, click maps and scroll maps should belong to the arsenal of every optimizer. However, you need statistically valid sample sizes to draw any conclusions.
So what’s the alternative? Use tools that use an algorithmic approach:
With these tools you can upload your screenshot and it will tell you something about your design that you can use. It’s not your actual users, but it’s something, and some people I trust swear on their accuracy.
User session replays
You don’t need a million visitors to record user sessions – this is almost like qualitative data. Use tools like Inspectlet, SessionCam, or Clicktale to record user sessions, and watch your actual visitors interact with your site.
Session replays are extremely useful for observing how people fill out forms on your site. You can configure event tracking for Google Analytics, but it won’t provide the level of insight that user session replay videos do.
One of our customers has an online resume building service. The process consists of 4 steps, and there was a huge drop-off in the first step. We watched videos to understand how people were filling out the form. We noticed the first step had too many form fields, and we saw that out of all the people who started filling out the form, the majority of users stopped at this question:
Personal references! The form asked for 3. Most people had none. So they abandoned the process. Solution: get rid of the references part!
Talk to your customers or prospects
You need to talk to people. You have only 10 customers? Great! No customers at all? No problem!
If you’re starting out, what you need is customer development, not conversion optimization. Food On The Table started with a target market assumption, and found a mom who plans meals and uses coupons, and spent three weeks shadowing her as she made lists and pushed her cart around the local supermarket. Her feedback helped create the first version of the website. Last spring they reached 1 million users. Qualitative research ftw!
If you have 10 or 45 customers, pick up the phone and call them. Don’t ask “how satisfied were you on the scale of 1 to 10”, ask questions that provide insight into their shopping process, their life, and how your product fits into their life.
Make your site blazing fast
Faster sites convert better. Here’s how to make your site faster.
Does your site work with every browser? Each version of Internet Explorer? If your answer is “I’m sure my developers tested it”, you’re probably losing money. You won’t believe how many conversions are lost due to poor cross-browser and cross-device compatibility issues.
Do this: walk through your site with every browser and every device (desktop, different tablets and smartphones) – go through the whole shopping process, and keep your eyes open for bugs and user experience issues. Make sure you fill out all the forms, click all the buttons and so on. Finding and fixing technical issues is a low hanging fruit that can bring forth considerable improvements in conversions.
Once you’re done, implement and watch the numbers
If you don’t have enough traffic to test, just roll out the changes (all at once) and observe the impact on your KPIs. Don’t be timid and just change 1 word somewhere. Implement all your hypotheses at once, or go for a radical change. Avoid doing meek changes and hoping for a 5% lift, go for a 30%, 50% or 150% lift – positive change big enough that you will notice right away on your bank account or in the number of incoming leads.
You have to be aware of possible seasonal impacts and changes in traffic sources. Then, segment the data in GA as much as you can.
Another way to do it is sequential testing. It’s when you run one treatment, then run another (for an equal period of time), and compare. This way you have double the traffic to each treatment, and it’s way faster than proper split testing.
Yes, it’s not scientific, it’s not really apples to apples comparison. But it can work. The key here is to set your test for a time that historically performs very evenly, and run “tests” for identical time periods (e.g. both tests Monday to Sunday).
This will not work (at all) if you have significant changes in your traffic sources, if you run ad campaigns, if it’s a holiday season, if you get a PR win and social media goes crazy about you. As soon as that happens, you get heavily skewed results and you have to start over.
Low traffic means you might not be able to run a/b tests, but you sure as hell can do conversion optimization. No excuses, get to work.
Join the conversation
Add your comment
Fantastic article and nice external resources.
What is a “low traffic” level ? How much visitors / months ?
Many thanks, I wish I could meet you in Boston…
Thanks. Low traffic = it takes forever (many many months) to run a split test until statistical significance.
If you split test 2 variations, get 100 people / day to the page and your control converts at ~2%, it will take close to a year to run 1 split test. That’s low traffic :)
Very great post as usual Peep!
I get around 12,000 unique visitors a month right now, which is not crazy traffic or anything, but good enough to run some tests.
I have been split testing my optins, but to get statistical significance faster, I have been doing what you just said by letting one form get all the traffic, looking at the conversion, then putting in a new one for about the same amount of views…and comparing the two instead of just doing an AB test.
That should be pretty accurate right?
I it is tough traffic, my site is a place where waiters submit funny stories etc, I am going to be developing a course for making bigger tips to try and monetize better…just added a feature box to top with email capture but just haven’t found a real winner yet.
Thanks for the amazing content, your blog could be sold as a product! Speaking of products….any further development on the course you were working on?
Consider one sold already!
Hey, with your traffic you should absolutely do proper split testing and not sequential testing, otherwise you really don’t know the actual impact of the changes and run the risk of using bad versions as its not apples to apples. There is no substitute for a/b testing. Sequential testing should only be used if your traffic is really low and you’re “testing” very different versions.
Thanks so much for the reply and advice! Will keep at it! Really appreciate the help….I thought that by putting in my email here I would get notified that you replied? I did not, otherwise I would have thanked you sooner! oh well.
BTW, have you ever considered going on Clarity.fm? (no affiliation)
Basically you set a rate and charge people per minute of your time….if you ever go on there please let me know as I would love to ask some questions of you but wouldn’t dream of asking them if you were not getting compensated.
Glad to hear course is coming out in October…thats when I am getting married…it shall be a wedding present to myself!
Oh, and the course is probably coming out end of October. Thanks for checking.
Great post, Peep.
You’re absolutely right! Sites with low traffic can take months to reach a conclusion. One site saw 100 visits per month to a page we were testing and it literally took 6 months before we could identify a conversion winner…
Having a basic CRO toolkit that includes usability testing, heat map analysis and customer surveys is more than enough to get started with the CRO process, all of which are pretty cheap ($100 per month?).
Then when you have a few business cases, you ask management for more pricier tools such as Clicktale.
Great post. 100% agree. The only issue is that people do sequential testing and are not aware of the factors that may skew the results, such as seasonality which you mentioned.
I agree, sequential testing should be avoided whenever possible, too many factors at play.
Interesting stuff, especially if you’re in a niche market in a small country like I am.
Even with great positions in search engines, the traffic for certain terms just isn’t large enough to do A/B tests.
I’ll definitely give sessioncam a go to, because with low traffic, it’s hard to justify the cost of Clicktale, which offers no entry level options (despite being a great solution)
If you want to do user session recordings, Inspectlet is the cheapest option. I do have doubts about the accuracy of their heat maps though.
I went with sessioncam eventually, as it worked out to be the best option, given the amount of traffic.
So far, pretty pleased with their support as well, as responses came quick and the account was created quickly (despite the original indication that it would take 24 hours or so)
Anyway, thanks for the suggestion of both, since I wasn’t even aware of these options.
I’ve just read a great post of guy who spent 4 years for a/b testing, and he got an amazing result from a very little change in detail. And now, your great post does help me to be motivated to take action immediately :). Thanks!!!
drifted here from google plus without expecting much, but what I saw is amazing. Would come very handy for a a very new site which I have launched. Thanks Peep
This is very nice explanation. Noting much but I totally agree that there are much things for solving the problem of traffic and getting the useful data. Your article solved out my queries. Thanks for sharing.
Hi Peep, great article.
What would you consider the ‘minimum’ visits for A/B testing, in other words what would you define as “very little traffice”?
We’ve always worked on the basis that 1,000 unique visits is sufficient for a statistical edge. Agree/disagree or that too arbitrary a question?
Visits matter little, 1000 is NOT valid sample size for most cases (unless you have extremely high conversions).
Ideally you call a test when you have 250 conversions per variation (=500 total conversions).
If your conversion rate is ~2%, then 1000 visits is like 20 conversions. VERY FAR from significance. So any “test” you called with 1000 visits is bogus.
You have to aim for 95% or more statistical significance. You can calculate the significance of your test with a tool such as this one: http://mystatscalc.com/
These are some very useful tips. And they make sense even to me, although I’m not very good at all this stuff. The “testing everything” part is especially understandable.
Thanks for the great article!!!
Comments are closed.