5 Usability Testing Mistakes You’re Probably Making

What are the mat common mistakes that test moderators make? We have observed usability tests moderated by consultants, in-house researcher, junior UX researchers and experienced practitioners and there are some common mistakes we come across time and again. These mistakes are like a rite of passage on the route to becoming a UX researcher, but even experienced practitioners aren’t immune from making them.

Moderating a usability test is full of bear traps. The moderator may fail ro set expectations (by reviewing the purpose of the test and describing the moderators role), forget to reassure the participant (“We’re not testing you’” ) or fail to check, for understanding (by asking the participant to repeat the task in his or her own words). Other common mistakes include asking leading or biased questions, and quizzing participants on how they would design the interface.

But there are five mistakes that we see usability test moderators make frequently and that eclipse all of these. They are:

1. Talking too much.
2. Explaining the design.
3. Answering questions.
4. Interviewing rather than testing.
5. Soliciting opinions and preferences.

1. Talking Too Much

When moderating a usability test, you need to fight against the tendency to talk too much. This can happen in two places, at the beginning of the test; and during the session itself.

It’s true that you need to provide an introduction to the session to put the participant at ease; and you also need to explain the kind of feedback that you want from the thinking aloud technique. But you shouldn’t go overboard in your introduction: Five minutes or so is usually enough.

Usably resting is about absolving participants while they carry out realist tasks. This means the golden rule is to shut up. Although moderators tell us they know this, we still  see many of them (even some experienced ones) failing to practice it. In the white heat of the test session, they can’t stop themselves from filling the silence.

This happens partly because people are not comfortable with silence and partly because there’s a misconception  that if the participant is’n’t speaking then you are not learning anything. But because you’re interested in participant behaviour, it’s fine to have periods of silence. Of course you want participant to think aloud but at the same time , you need to allow participants space to read, make judgements and generally think about what they are doing.

You can avoid this trap by learning to embrace the silence. Ask participants to do the task. Then shut up, observe, and listen to what they say. if you feel the urge to speak, use a phrase like, “Tell me more about that.” If you force yourself to use the same stock phrase, and none other, it will help you stay silent (you’ll sound stupid if you use it incessantly to fill the silence and you won’t do too much damage because you’ll encourage the participant to talk.

2. Explaining the Design

If you ever find yourself saying to a test participant, “What the developers are trying to do here is..,” or “The reason they designed it this way is because…,” or “What you don’t understand is..,” then you should slap your-self. When you explain the design of your product to a test participant, it causes two problems.

First, you’re no longer able to find out how someone will really behave when they first encounter the design. This is because you’ve given the participant some background information that real users probably won’t have.

And second, even if you were never involved in the design of the product, you affiliate yourself with it. Because what the participant hears isn’t an explanation of the product but a defence of the product. This prevents you being seen as a neutral observer and makes it more likely that participants will self-censor their comments.

The point where this problem occurs most frequently is during the test tasks themselves. The participant may use the product the “wrong” way and the moderator feels the need to explain how to use it “properly.” Or the participant may be critical of something in the interface, and the moderator feels the urge to defend the design with a phrase like, “The development team thought about doing it that way, but…” Or the participant may completely misunderstand something in the interface, at which point  the moderator will want to correct the participant’s mis-understanding. In particularly bad situations, this moderating style risks turning the usability test into a coaching session, or even an argument.

Believe me when i say  that no usability test moderator ever won an argument with a participant.

If you ever feel the urge to explain the interface or use a phrase like, “yes but …” then instead say, “Tell me what you are doing right now.” You will then get behind the behaviour without influencing it too much. If you really, really want to explain how to use the product or correct any misconceptions, then wait until the end of session, once participants have tried it without your help.

3. Answering Questions

Here’s another trap we see moderators walk into, is like watching a slow: motion replay of a dog chasing a stick over a cliff. The participant sets the trap and the moderator runs towards it.

Like most traps, it seems fairly innocuous. The participant simply asks a question.

Now, participant questions are like gold dust. You want participants to ask questions because this indicates they are experiencing a problem with the product: They’re not sure how to proceed, so they ask you.

Gold dust, but not gold.

You find the gold by observing how the participant answers their question: What do they do to solve the problem? Do they find it easy to fix or do they consistently take the wrong path? It’s their behaviour that helps you distinguish a low priority problem from a critical one. This means the route to the gold is to refuse to answer the question.

But to any normal human being, refusing to answer a question is alien. From childhood, were conditioned to think that ignoring a question makes us appear either rude or stupid. That’s why so many test moderators walk blindly into the trap of answering participants’ questions.

Here’ the way to fix this in your own practice. First, in your preamble, tell participants you want them to ask questions but you won’t answer, because you want the session to be realistic. Use a phrase like, “Just do what you would do if I wasn’t here?” This then gives you permission not to answer any questions you’re asked.

Then, when the inevitable question comes at you during the session, use the boomerang technique Answer the question with a question. So, if the participant asks, “How do I get back to the beginning?,”” you respond “How do you think you get back to the beginning?” If the participant asks”whereabouts  is the registration form?,” you reply: “Where would you look for it?”

4. Interviewing Rather than Testing

If you’re invested time in getting participants to attend your session, it makes sense to get as much out of them as possible. So you should certainly run a pre-test interview with participants before they start the test tasks to find out more about them and their relevant goals. But while the participant carries test-you’re an observer.

Here is a common situation that causes a usability test to degrade into a interview: When the development team don’t know much about users, the team may not have done any field research in the past  and want to milk this session for all its worth. This shows itself when the participant is interrupted mid-task and asked questions about the way they do this task at home. Or when the marketing lead asks you to shoe-horn in a shopping list of questions during a task. As a consequence, the research falls between two stools: It’s neither an interview nor a usability test.

Another situation where this can happen is when you have a particularly loquacious participant who wants to engage the moderator in conversation, rather than do the tasks. The participant will continue to look over to the moderator for reassurance and try to make eye contact.

The best approach is to prevent this problem from happening in the first place. Adjust your body language to be more of an observer than an inter-viewer. Position yourself so you are behind and to one side of the participant.

If you sense the participant looking toward you, pretend to take notes and decline the offer of eye contact.

Also make it clear to the development team that you’ll run a post-test interview to get an overall assessment and encourage comments regarding topics not raised during the session, and that’s where you’ll cover their shopping list of questions.

5. Soliciting Opinions and Preferences

This final mistake is one we see often in people who are new to moderating a usability test. This is because they have confused usability testing with market research. They think their role is to solicit opinions rather than to observe behaviour.

The way this manifests it self in a rest is the moderator will ask the participant to compare different designs to see which one they prefer, or they will continually ask the participant if they like or dislike some design feature.

Usability resting isn’t about finding out what users like, but rather what works best for them.

How to Continuously Improve as a Test Moderator

These mistakes almost always occur in novice test moderators as they earn their spurs. But even experienced test moderators make these kinds of mistake during a usability test. The best way to avoid mistakes is to continuously reflect on your own moderating skills. After each usability test, look back over the recordings, especially sessions that you feel went particularly well or badly. Make it part of your personal development to identify three things you can build on or that you could have done better.