The Scrappy Entrepreneur’s Guide to Usability Testing

The Scrappy Entrepreneur's Guide to Usability Testing

Usability testing is important, but when you’re juggling tons of other tasks – acquisition, hiring, and whatever other fires you need to put out daily – it can be thrown on the back-burner.

“I don’t have the luxury of focusing on UX right now,” you may say to yourself.

But this guide will prove that usability testing need not be time consuming, expensive, or obstructive to any other priorities you may have. In fact, you can run quite lean and get game-changing usability metrics pretty quickly.

Before I get into how to do usability testing, let me start by saying why I think it’s wrong to view UX as a luxury.

Why Do Usability Testing, Anyway?

You might think UX, or User Experience, is about making things pretty or “delightful” (how frivolous). In fact, UX is really about good business. Done right, optimization of the user experience:

That list could be longer, but the point is, optimizing the user experience is all about increasing conversions. With a good UX, you help yourself by helping your users – by making it really easy for them to achieve their goals.

Usability testing is one of the main ways to learn about the user experience on your website or app. This method of research puts you right into the shoes of your users so you can see where they get frustrated, lost, or confused, and then improve the UX based on what you find.

“Ok, but I don’t know anything about UX or usability testing.”

Optimizing the user experience is a great idea, but only if it’s not going to cost an arm and a leg. For a scrappy entrepreneur like you, hiring a UX expert is over the budget, and you can’t just do it yourself, right?


Here’s the secret: with a little bit of guidance and a little bit more common sense, you can do it all on your own. The cost can be low and the time commitment reasonable.

User testing isn’t as difficult to juggle as you’d imagine

This guide will walk you through the 6 key elements of doing usability testing, with all the time-saving and cost-reducing tips that a scrappy entrepreneur like yourself needs to know.

Section 1: Where to get testers

Recruiting people to test your website or app is the first piece of the puzzle. Ideally, your testers should be people who are representative of your company’s customers. The closer they are to your target market, the more accurate and useful their feedback will be.

  • Actual customers: You can ask actual customers to take your test.
    • Pros: Definitely representative.
    • Cons: Already familiar with your website, so they have bias. Great option for testing new things no one’s seen yet.
  • Craigslist/TaskRabbit: Some companies find representative testers by posting on sites like Craigslist or TaskRabbit.
    • Pros: Fresh eyes, wider reach.
    • Cons: May require higher compensation. Greater variability in quality. Least amount of mutual trust.
  • Coffeeshop: Approach people in a cafe or similar venue and ask them to look at your site and give feedback (perhaps in exchange for a coffee or a muffin).
    • Pros: Least effort, pretty cheap.
    • Cons: Least likely to be representative, so their feedback may not be all that valuable.
  • User testing companies: Most remote testing services like ours have a pool of testers and some level of demographic sorting.
    • Pros: Wide reach, filtering capabilities, no work for you.
    • Cons: May be hard to find specialized target groups.

“When you create a product, it’s intended for a specific type of person. So when you do research, you want to engage people who are as similar to your target user or audience as possible.”Sarah Doody

Section 2: How to run the test

The next component is deciding how to run your test. This will depend on whether or not you want to moderate the test.

Moderated testing is when you sit in on the session with the tester, talking directly to them to give instructions and ask questions.

Moderated user testing with GoToMeeting Image Source

Moderation gives you more control over the flow of the test and lets you react and adapt as it happens. You can ask follow-up questions or give clarifications to the tester depending on what happens.

  • For moderated testing, you can use any video-calling tool with screenshare features. Skype is a good free one; Google Hangouts is another.
  • To record your test sessions and re-view them later, you’ll need a paid service like GoToMeeting or Zoom.

Unmoderated testing is when the tester performs the test on their own by following a prepared instruction list, without outside interference.

Unmoderated tests more closely resemble a real-life user session on your store, as the tester is forced to be wholly self-reliant. It also eliminates psychological biases caused by the presence of a moderator.

  • Online services allow you to create a task-based instruction list for your test and then send it out for testers to perform.

If you don’t have much experience moderating user tests, unmoderated may be a better option. Moderating requires lots of restraint, totally neutral reactions and word choices, and skill in getting the test to unfold in the right way.

Section 3: How to write the test script

Whether you go moderated or unmoderated, you’ll need a script to provide the framework of the test. Typically this is a series of tasks, sub-goals that gradually lead the testers from start to finish in a logical and natural way.

  1. Your script should be based on real user pathways. Identify common flows that your users follow (look at your web analytics for this), and translate those pathways into step-by-step tasks.
  2. Give the tester a scenario they can relate to – a story to step into. Who are they, why are they here, what are they doing? Start the test with a bit of background.
  3. Break it down into bite-size pieces. Let testers focus on just one thing at a time.
  4. Write goals, not instructions. In other words, tell the tester what objective to work towards, then see how they do it – telling them “Go here” or “Click this” defeats the purpose.
  5. Be careful of the language you use. Avoid jargon or words and phrases specific to your company. Write plainly, in ways that a user in your target audience will understand.

Extra tip: Include an impression test or “five-second test” at the beginning to gauge people’s first impressions and test your brand imagery, communication, and home page effectiveness.

Section 4: What data to collect

With usability testing, the most important thing is the screen and voice of the tester. Screen capture provides a window right into the user’s experience so you can see what they’re seeing and watch what they do in every situation.

Voice recording is the more valuable part, though, because it contains the why – Why did they click this button or that link? Why did they get stuck here? Why was this page so frustrating? Being able to hear users explain why in their own words is the biggest benefit of this kind of research.

There are many other kinds of data that you may want to collect in addition. Some can be gathered just through analysis of your videos:

  • Time to complete a task
  • Proportion of testers who complete a task
  • Number of clicks

Other types of data require you to administer additional surveys, like:

  • SUS, PSSUQ, or SUPR-Q to measure overall system usability
  • SEQ to measure the usability of a task
  • NPS to measure users’ attitude towards your company
  • Written response post-test surveys

Read more: 8 ways to measure satisfaction (and improve UX)

All of these metrics have their uses, depending on the type of interface you’re testing, who you’re testing with, and the purposes of your research. It’s good to collect at least 2 or 3 of these metrics when you run a usability testing study so you can use them as a benchmark for next time.

Section 5: How many tests to run

The most common sample size for a user testing study is 5 people.

Well-known UX research firm NNGroup has posited that 5 testers is where you get the most bang for your buck due to diminishing marginal returns. A sample size of 5 is very manageable from a cost and time perspective, and will uncover about 80% of usability problems.

If collecting quantitative metrics (like the ones mentioned in Section 4) is a big deal for you, you’ll need more than 5 testers. To get reliable data generally requires at least 20 testers, so you had best find a cheap (or free) way to recruit!

However, even with a smaller sample size your quantitative data can be useful. Task usability scores can be compared across a single person’s test to see which part was the hardest; system usability scores can be compared between testers to see who had the worst experience.

If you’re really in a rush, try running 3 tests. Be careful with less than that: any 1 tester could be way off the mark, so you want a few to cross-reference against.

Section 6: How to analyze the results

We’ve run a lot of usability studies on many different kinds of companies at TryMyUI. While everyone has different strategies for tackling their results, here are our top tips (and tips we’ve heard from our customers):

Look for the discrepancies between how you intended things to be used, and how people actually use it. What do they do that’s unexpected? Do they struggle to find things you thought were obvious? Do they misinterpret the words and labels you’ve used?

First impressions are make-it-or-break-it for many websites and apps. Pay attention to what people say, do, and look at when they first see yours. What do they notice first? What words do they use to describe it? Do they understand what your company offers? Do they identify with it as a brand?

First impressions matter in user testing

Every time you note down an issue, list 1 or 2 broad topics that it relates to: for example, “search,” “navigation,” “checkout,” “language,” or “images.” Then, at the end, you can tally them up to see where the most problems occurred.

Some people use post-it notes to write down issues, then sort the notes at the end as a visual aid to see big groupings.

Watch for positive moments as well as critiques: you want to know what you’re doing right, so (1) you can accentuate it and (2) you don’t change or remove something your users like.

Save key video moments. Short, succinct clips straight from the user’s mouth can be very powerful for demonstrating a point to your partners or investors.

Quantitative data can help you before you even start watching the videos. If you can identify which tasks were the hardest, or which users had the worst experience, you can cut straight to the chase and watch those first.

If someone else is available to look at the results, have them review the videos too and then compare notes. This is a great way to account for your own biases and blind spots and make sure you didn’t miss anything.


And that’s the guide! Congrats, you are now ready to run your first usability testing study.

Even a little bit of research helps, but testing regularly – once a month, or once a quarter, for example – is a great way to always be optimizing and keep making it easier for visitors to become customers. Make sure to collect some key metrics before and after you change your design to measure the effectiveness of your optimizations.

Lastly, remember that a lot of this is just common sense. Don’t overthink things, usability testing can be pretty simple. It’s all about getting out of your own head and seeing the user’s point of view, and if you take that goal seriously, and go into it with that mindset, and are open to hearing criticism and negative feedback and making changes to your own creation, you’ll see it pay off. That shouldn’t be too hard for you – you’re an entrepreneur!

Related Posts

Join the conversation Add your comment

  1. Awesome post. We’ve always used current customers or fans for some usability testing (even if they haven’t been accepted into the platform yet)! Thinking about using some paid usability testing sites like you’ve mentioned. I’ll let you know how it goes if we decide to test it!

    1. Thanks, Aaron – I hope when you do, this guide comes in handy!

  2. Hello,

    Great Information your have shared. Usability tetsing is the bets option to find and resolve the website issues on the basis of feedback given by real time users. It may help business owners a lot.


Comments are closed.

Current article:

The Scrappy Entrepreneur’s Guide to Usability Testing