5 Weeks of Cognitive Walkthroughs

I just completed over a month of back to back usability testing sessions for different clients along the eastern seaboard. This consecutive approach to cognitive walkthroughs was exhausting. I’m literally sitting on the plane about to pass out–thankful I’ve finally got a day off to relax tomorrow.

As a UX consultant, I often find myself in different offices watching people use software I helped develop. There is so much you can learn by simply watching people. Some are resistant to change, some embrace it, some are computer savvy, and some have limited experience. Regardless, sampling a variety of users is important because it allows you to collect constructive feedback from all different perspectives.

Since I haven’t blogged in a while, I thought I’d share what I’ve “learned and confirmed” in the last five weeks:

• After observing as few as 5 users, you’ve pretty much seen about 80 percent of your usability issues. I’ve experienced this several times before, but when you have an inventory of hundreds of users, it doesn’t take long before things get repetitive.
• You’re not testing the user, you’re testing the system. Things are never their fault. You need to remind your users of this.
• Unless you can do something with what you’ve learned, it’s just information. You need time to reflect and act. Once you’ve spent three days with a particular set of users you are wasting time and money.
• Some users will be brutally honest with you, some will be quiet. There’s also one guy in every office who is a total jerk-face. Don’t let them get in the way of doing your job.
• Your findings will hold a lot more weight if you’re a 3rd party. Remind them you have no role in the development of the software. If you do have a role, don’t tell the user who you work for directly. Just let them know you’re simply there to test the software “Acme Inc.” built. If things go really well, feel free to tell them who you work for after the observation is over. You never want to risk it by revealing who you are.
• If your users needed to reference training or an online help system, you’re not doing a very good job as a designer. There is no such thing as a “training issue” in regards to usability. Did you need training to use popular internet applications such as YouTube, Facebook, or Twitter? Usability issues are just as important as system issues. Document everything. UX design minimizes the need for training.
• You need a control mechanism. A survey should back up your observations. If it doesn’t, you’re not listening hard enough.
• One opinion or frustration does not constitute a design makeover. Look for the trends. Follow the famous 80/20 rule. If most of your users get stuck somewhere, it’s time to rethink your design.
• You’re not there to train, you are a fly on the wall, but don’t let the user get upset. Once you’ve got it document, help them out so you can move on.

Updated June 13th

• Stay off of your blackberry. There is nothing that drives people more crazy than someone who is wasting their time and not even giving them their undivided attention.
• Maximum one observer/per user, period. I’m serious here. There should be one person facilitating a cognitive walkthrough. If you absolutely must have another person, that person should stay quiet and take notes. Risking the user feeling uncomfortable will impact your findings and sway the results (possibly in a negative way).