Usability testing is the black horse of boosting conversions. If your site is difficult to use or hard to understand (means it has usability problems), it will result in poor conversions.
Website usability testing helps you identify all the main sources of friction, and thus boost conversions. The premise is simple: observe real users use and interact with your website, pay attention to their experiences and try to spot patterns. Once you have the results, iterate your site design and copy to minimize any barriers to accomplishing their on-site goals. This, in turn, will also lead to increases in conversions.
Jakob Nielsen was able to show in an analysis that the mean increase of conversion rates with usability optimization was 87% (he considered 66 studies with before and after measurements in which the usability of websites was optimized with the help of usability tests). On average, the conversion rate could almost be doubled.
While that analysis was done ~5 years ago and the average level of website usability has gone up since then, the point remains valid.
Table of contents
The importance of website usability testing
Usability testing involves setting a series of tasks for people to complete and noting down any problems that they encounter. It really is as simple as that.
Or as Steve Krug, the author of “Don’t Make Me Think” puts it:
Usability really just means making sure that something works well: that a person of average (or even below average) ability and experience can use the thing – whether it’s a web site, remote control, or revolving door – for its intended purpose without getting hopelessly frustrated.
User testing gives you direct input on how real users use your site. You may have designed what you believe is the best user experience in the world, but watching real people interact with your site is often a humbling experience. Because you are not your user.
Users think differently
The thing is, your customers see your website through their eyes, not yours. You see the world as you are, and you spend way much time on your site. And that’s where usability testing comes into play – giving you direct access into what users are thinking when they are browsing your site. The information you get is pure gold.
Steve Blank, a Silicon Valley mentor and serial entrepreneur, recognized for developing the customer development methodology, says it best:
No plan survives first contact with customers
Sure, he was talking about business models and business plans, but it applies to websites as well. Would you rather find out that your website is fatally flawed before or after the launch?
With usability testing, you will find out most of the flaws before and that is crucial. Heck, you can find out before it’s even built by incorporating usability testing into your design phase.
Goal of usability testing
A major goal of any usability testing is to find out whether users can easily find and understand the information they want or need, and whether they can complete tasks at hand without much effort.
Once you’ve identified any and all of the friction points, you can fix the issues and improve the user experience for your website users.
Usability vs A/B testing
Usability testing is different from A/B testing (but you might do usability testing for different versions of your site).
A/B testing is done on a live website with real visitors who have no idea that they are part of a test. Usability testing on the other hand usually involves people observing recruited testers completing a given set of tasks on the website. User observation can take place in-person or it can be remote, online – recording users’ screen and voice as they comment their thought process out loud.
The beauty of usability testing is that the site doesn’t need to be online yet – you can test whenever you want and whatever you want at any stage of the design process.
One more difference is the sample size: A/B testing involves hundreds if not thousands of people while usability testing can be done with 10 people or less.
Additional insights
Usability testing can provide additional insights that A/B testing cannot such as emotional responses as well as fast detection of issues. A/B testing will give you statistical data that X is working better than Y, while usability testing will tell you WHY something doesn’t work, HOW they would like to use the site and WHAT exactly is causing friction. It can be a much more insightful learning process.
Bottom line: Usability testing is different from A/B testing, but not a replacement for it. Use both. Conduct usability to identify any issues, fix them and test the impact. Here’s a post on how to use usability testing and A/B testing hand in hand.
3 steps to running usability tests
Pretty much anyone can run their own usability tests. Here’s how.
Step 1: planning
The plan should cover how the test is run, what are you trying to figure out and how are you going to do that.
Elements of the plan
Based on Usability.gov’s excellent materials, here are the elements that a plan should cover:
- Scope. What exactly are you testing? The whole site, only part(s) of it? The navigation, navigation and content, just content? In this step, you indicate what exactly are you going to test.
- Purpose. Identify the concerns, questions, goals and hypothesis you are going to test. These could include questions like can users easily find info about product X? Or is our store locator feature easy to use? Or is the check-out procedure easy to use?
- Schedule and location. When and where are you going to be running the test? In-person or remote/online?
- Participants. How many people are participating in testing, do they have to be from a certain demographic?
- Scenarios. How many tasks and scenarios are your participants going to go through and what are they exactly? This is where you will formulate concrete questions and actions like find product X and fill out the form for ordering etc.
Using these elements as a baseline you are well on your way to successfully running usability tests.
Recruiting testers
Word of advice: only test with people who match your ideal customer profile. I’ve done plenty of tests with testers recruited via usertesting.com who couldn’t figure out what the site was about since they were not the target group. They ended up giving totally random and useless feedback. (The site in question was an online software that helps bloggers build courses online – and testers didn’t know what information products or ebooks were).
A lot of the testers these remote testing services provide are stay-at-home moms and other unemployed folk trying to make an extra buck. Might not be your target market.
Recruiting any random person as a tester might work if you sell shoes and socks to everyone, but if your product / service is somewhat specialized, you can’t do that. You want to test if your copy and layout work for the people you’re actually trying to sell to. No point asking feedback from computer-illiterate grandmas about your bleeding-edge cloud hosting service.
The best way to recruit testers for specialized product/service is from your own network – your blog readers, newsletter subscribers, Twitter followers, friends of friends. People that match your customer profile.
If you’re recruiting your own testers, always reward them for their time. You can give free products/services, coupons or $25 Amazon gift cards. They’re providing a very valuable service.
Step 2: running the test
There are 2 ways how you can run usability tests – either in-person over the shoulder or using remote testing tools.
1) Over the shoulder testing
The most insightful way to run usability tests is to conduct them in-person, using the so-called “over the shoulder” usability testing.
All you need is an empty room, a working computer (with screen recording turned on – you WILL forget) and a notepad + pen for writing down your observations. You give the test subjects a list of tasks you want them to do on the site (browse around, look for product X, find something you like and buy it etc), and observe them do it. Testers have to comment on everything they do and think out loud. If they’re having trouble understanding a task, you can clarify. If you have something extra to ask, you’re right there.
This can also be done in a more formal lab setting with cameras that record the facial expressions of the participant etc.
The downside of this form of usability testing is that people might change their behavior when they know they are being watched. Another downside is that arranging a mutually convenient time and place can be tedious.
2) Unmoderated remote testing
Another form of usability testing is doing it remotely via online tools like UserTesting.com, Validately, TryMyUI and others. The only real difference with over the shoulder testing is that it doesn’t have a physical facilitator (you) watching over things and thus it can save you valuable time. It’s unmoderated.
Remote testing works by recording audio and what’s happening on your screen via screen capture software (all of which you don’t need to worry about, it comes with the service). Usually, they come with a ~20 minute limit to video length, so it has to be possible to complete all the tasks within that time frame.
The facilitation is done through pre-loaded prompts that appear over the top of the website being tested. In the end you will get a video recording where you can see and hear exactly what was going on.
The lack of an actual person looking over things creates a situation where the quality of the feedback can’t be controlled and may cause problems. It’s not uncommon that a task is communicated ambiguously, so the user spends 10 minutes doing stuff that is not important. Or they run into unforeseen technical issues that are also not relevant to the test.
The good part is that it’s relatively cheap, fast and easy. After you set up the test in one of the above-mentioned tools, you just send the link to test subjects and they can do the test whenever – you don’t need to be there. That’s especially awesome if you have a global user base and would otherwise have to work hard to find a suitable time across time zones.
Step 3: analyzing results
After the tests have been done, gather all the data and write down the main points. You need to be able to compare how each of the testers did on each specific task – was it smooth, what kind of problems did they run into, any insightful comments that they made.
Be ready to be surprised – usability testing with real people brings out things that you would never have guessed yourself.
Draw conclusions, create a list of identified usability problems and figure out how to solve them.
Usability testing in action
Want to see what a recorded usability test looks like? This one comes courtesy of Guerrillo.com and it’s run on Vibram Five Fingers homepage. The video, found here, runs for around 14 minutes.
In it, participants have 4 tasks that need to be completed. The First task is finding your shoe size, second is finding the best Vibram for running, the third one is about finding a pair good for traveling, and the last task is finding a local store where the shoes could be purchased.
What can we learn from it?
The test itself is fairly simple and involves only 4 tasks, yet it manages to give lots of input on how real users see and discover content and use their site.
It gives insights into ease of navigation of the site – how easy it is to find certain pages, how long it takes? Additionally, it manages to assess understanding of the information given – everything for finding the best shoe size is there but how will users use it, can they understand the information given? Or running shoes, will they be able to find?
The last point manages to assess the store locator feature of the site. Someone has been putting a lot of work and effort into building that one feature and it’s assumed that it’s easy to use and easy to find info from – is it really the case?
I’m more than sure that people who designed the page and ran the test didn’t anticipate that finding shoes for running would take that long, after all, there is a link in the navigation bar for “Barefoot Sports”, but it did. Without usability testing, it would have been hard to spot.
Case studies
Kirklees (a region in the UK) had developed a site for a visitor of the Kirklees area. Their goal for testing was to better understand how useful and usable their site was in real life. They chose 9 testers from different age groups and ethnic backgrounds.
The tests were conducted in different settings – both observed (over the shoulder) and unobserved methods were used to get all possible sides of the story.
The research identified around 70 usability and content issues. During the facilitated discussion, Kirklees council staff gained a shared understanding of these issues, and actions were identified that could be implemented within the project scope. As a result, they were able to act quickly to improve the site.
My friends over at Conversion Rate Experts worked on a UK-based travel agency site and among other things turned to usability to boost conversions. Here’s what they had to say:
In our usability tests, we discovered that visitors were often confused about the prices. Prospects weren’t sure if the prices were per person, if they were per night, or if flights and hotels were included. Ironically, the prices were so low that the company was harming its own conversion rate.
By clarifying the pricing, and what it included, we increased the conversion rate by 19%, adding an amazing £4 million to its annual revenue.
Not bad.
Some tips
1) Number of participants
According to research done by Jakob Nielsen (known as the godfather of web usability, coined this term in a 1994 paper) best results come from testing with no more than 5 users and running as many small tests as possible instead of one big elaborate one.
The logic being that as you add more users, you learn less and less because you’ll keep seeing the same things over and over again.
“After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.” says Nielsen.
First 5 will tell you your biggest issues, testing with 10 users you will find basically all of the problems.
2) Don’t jump to conclusions
People are different and unique in their own way. It’s essential to keep that in mind when doing usability testing – just because one or two participants had trouble with something doesn’t mean that you should jump into changing everything.
Make changes only when there is a clear trend among testers, and put their challenges in context (who these people were, skill level etc).
3) Record your testing session
There is only so much you can write down at any one time, so having a recording of the session is a godsend.
4) Test early and often
It’s far easier to make small incremental changes throughout the lifespan of your site or product than it is to change one huge lump at the end of it. Much cheaper too.
5) Be open-minded
Don’t go into usability testing expecting a specific outcome, and only looking for evidence that supports your hypothesis. Try to keep an open mind, and listen to your users’ feedback. See what the results say, do your best to learn from them, and your site will be that much better for it.
*Data for this list came from here, here plus our own field testing results.
Conclusions
Usability testing is a simple, yet extremely powerful way to get valuable insight into how real users interact with your site and the issues they run into. You need to know this stuff, and fix it.
It’s not expensive or complicated. The ROI on this stuff is more than excellent.
Great post!
Usability testing is a great method to find big conversion killers, that you thought your site would never have :) I’ve used it, like you, as first step of finding problems, and then used A/B-testing to controll the changes. Great method to use alongside split testing.
Thanks for sharing!
Bloody good work there, Peep! You took the words right out of my mouth :)
;)
Great Detailed Post with lots of examples as usual..
Great usability test video too…!
Bookmarked this article..
Thanks again for sharing.. !
Some great tips here! We used usertesting.com with a client to test their checkout area and since this they have decided to stop requesting user’s mobile phone numbers as users were asking too many questions about why they needed it and causing abandoned carts. Could this be extended to mobile optimised versions of websites?
For sure. You can do user testing on mobile devices, but then in person – not aware of screencasting tools for mobiles yet. You can also do swipe tracking with http://heatdata.com/
Love the usability.gov link, the US gov telling us how to make useful sites, but with a totally crap outdated site lol.
Fascinating Peep. Your comment about over-the-shoulder testing is an interesting one. It may, in part, be influenced by what you tell the person you’re doing while they test.
The more innocuous you can make your presence feel (“don’t worry, I’m not watching you, I doing (something unrelated)” etc), the more ‘natural’ the results are likely to be, although there will always be some influence I guess.
Peep, this post is filled with tons of great information! Thank you so much for taking the time to write it.
I especially like the point you made about testing sites before you put them online. I started doing this about 6 months ago and have saved myself so much time. I’ll generally mock something up with Balsamiq, throw it online with Invision, and then get TONS of great feedback before I spend the time to markup the site.
Also, since I work at UserTesting.com, I wanted to point out that with our Enterprise platform you’ll have a much easier time targeting the specific audience you’re looking for. We have a lot of advanced features (moderated intercepts, advanced demographics, recruit from sampling companies) that aren’t a part of our pay-as-you-go product.
Keep up the great work Peep! I look forward to your next post.
Great post, glad I read it! Here are some thoughts on step 3 – Analyzing Results http://usabilitylab.walkme.com/main-usability-analysis-methods-to-use/, hope you like it :)
really useful post! I hope to increase my conversion rate, because it’s really low now :S