How to do usability testing.

@pacharrin on Twitter asks:

“I’m preparing my first usability test. Any recommendations?”—well I’m so glad you asked this question.

I’m going to give you the “low-down” on how to do usability testing that is both effective and eye-opening. By the way, this article is about doing a very small qualitative study, so we’ll save card sorting/eye tracking/etc for another day. Here are your action steps to testing software or a website for usability:

Like this article so far? Click here to tweet about this on Twitter.

1. Assemble Scenarios. Create a list of scenarios and prioritize them. It is very possible that you may not get through all of them. I recommend no more than 8 or 9 for a one hour study. You don’t want to have too many and your user get fatigued.
2. Assemble interview questions. What kind of questions will give your test results perspective? You can ask these questions before, during, or after the study.
3. Assemble a survey. This can be sent to the user via email before or after the study is completed using a service like Survey Monkey.
4. Get “buy-in” from upper management. You need to make it sound like it’s their idea. It’s so important to create an environment where everybody is thinking about the users and this is driven from data—not ego. Use the arguments “you can’t hold me accountable for usability if we don’t do usability testing regularly” and “if we’re not making decisions based on data, we’re just guessing!”.
5. Recruit Participants and make them sign waivers beforehand. Have a list of trigger questions that will disqualify them. You want to eliminate participants that may skew your results. You want to recruit 5-6 users and also try to recruit 1-2 backups in case some of your users don’t show up. By the way, someone almost always bails!
6. Choose testing style. Are you doing lab testing or field testing (ethnographic)? If you’re doing lab testing, make sure the lab has an observation room in case stakeholders want to observe. Regardless of the testing type, make sure that there are no more than 2 people in the room with the user (one person to facilitate and one person to take notes). Often times, management will want to sit in on these things—fight it if you can.
7. Figure out what equipment you’re going to use and keep it consistent for each user. I prefer a simple Windows 7 machine with Internet Explorer or Firefox. I recommend installing Morae to do picture-in-picture video recording. If you’re doing this in a lab, they may have the configuration available for you. Another option, if you’re using a Mac, is to try Silverback (much cheaper than Morae).
8. On the day of the study, make sure that refreshments and snacks are available. If you’re budget is very small, just have bottled water on hand. Your users will appreciate the gesture.
9. When the user arrives, kindly great them and thank them for participating in the study. Inform them what you are doing, that they’re being recorded, and that their contributions are going to be used to improve your product.
10. Begin with a few interview questions to get the user warmed up. Instruct the user to “think out loud” so you know what they’re thinking as they move through each task.
11. Start with a simple scenario, like “Navigate to Acme’s home page” then gradually move on to harder scenarios. Pay very close attention to what they’re looking at, their facial expressions, their complaints, etc. This is where it’s nice to have one person who’s role is strictly to take notes so you can focus on the subtle things.
12. Have your note taker discreetly record the time it takes to do each scenario. It isn’t a good idea to tell the user that you’re recording the time—and it doesn’t have to be too scientific, just glance at your watch.
13. If the user asks you a question, use “the echo effect” and repeat the question back to the user. Example: “How do you search for keywords?” reply with “How, DO YOU THINK you would search for keywords?” if the user doesn’t know, say “How do you think search should work?”. Remember, you’re not there to give the user training, you’re there to uncover usability problems so they don’t need training.
14. Make sure the user tells you when they would have normally given up on a task. It doesn’t mean you have to stop the scenario, but it’s important to indicate the timing in your report. If you strongly feel the task cannot be completed, definitely move on to the next task—but don’t give up too soon. There are often many navigation and search paths, you never know which one your user may discover.
15. When the study is over, pay your user an honorarium or gratuity for their support. Anywhere from $50-$100 for normal users. If the user is a subject matter expert (like a doctor or lawyer), it’s customary to pay them more their contribution.
16. Send a survey out to each of your users. Sometimes, I like to send out a thank you card in the mail as well.
17. Assemble all of your data together and create a report. Annotate the number of times a problem occurred and indicate whether or not a problem is a trend or not.