Moderated usability tests are the cheapest, fastest way to iterate on your product. You learn what not to build before even a single line of code is written, saving your team weeks of engineering time. In case you already have a product up and running, these tests let you know why your user conversion rate is messing up, where the user is rage clicking, etc.
As a bonus, you get to see users react in the most unexpected ways to your designs — confirming that people are not so predictable after all.
But like with most things in a business, decisions often comes down to 💰 at the end of the day. So here are some examples of how moderated usability tests, or the lack-thereof has impacted real products and companies:
Despite the obvious value of moderated usability tests, many teams struggle to conduct them because they don’t know how to do them well. One of the most common statements I hear from PMs and Designers is, “I’m not sure which questions to ask”.
This article will give you an overview of:
The contents of this article are equally applicable to remote usability tests and in-person usability tests.
First thing’s first — here are some basic guidelines to help you run a successful usability test.
A moderated usability test guages—you guessed it—the usability of your product.
You conduct a moderated usability test by giving participants a prototype or a product and asking them to complete one or more "tasks" on it. These tasks are representative of user flows you'd like your users to understand, without external intervention.
During the usability test you will observe your users and identify areas where they get confused or frustrated with your product. This helps you identify ways to improve the usability of your product.
Give your user a task that is representative of what they might actually be doing on your software. For example, when we were running usability tests for a time-stamped note-taking feature at Looppanel, we asked users to take notes during a mock user interview (a user interview inside the real user interview — meta, I know!).
With a usability test you’re trying to simulate how your user would feel the first time they interact with your product. What you want to know is — what’s going on in this person’s mind? How are they reacting to the product the first time they see it? Are they able to navigate through it successfully?
Since we don’t live in a world where you can read someone’s mind (thank god!), we do the next best thing with moderated usability tests — ask users to Think Out Loud. It is exactly what it sounds like — users stream-of-consciousness share all their thoughts, feelings, questions as they’re navigating through the prototype.
Did they look at a button on the right-hand-side? Why? What did they expect it to do? You really want All. The. Details.
For a detailed look at how the Think Out Loud (aka Think Aloud) Protocol works, check out this article.
You’re running a moderated usability test so you can understand the root of your user’s confusion and what their true needs and expectations are. You won’t get to the root the first time you ask them a question, so keep probing with Why until you hit the depth of the reason.
For example, when I asked users about emojis we were testing in our note-taking feature, many participants responded that they found them “confusing”. Although that’s insightful, if I don’t understand why they found them confusing, I don’t know whether to replace those emojis with new ones, with text, or remove them altogether.
By probing with ‘Why’ we found that emojis were open to interpretation for users — they weren’t sure what ⭐ or ✅ should mean. Given the rapid, time-constrained nature of taking notes during a call, they often chose to skip using the emojis altogether because it simply took too long to figure what to do with them.
Knowing why they were confused allowed us to step back and remove emojis from the note-taking view altogether, replacing them with a simple bookmark to time-stamp key moments that didn’t require too much hemming and hawing to figure out.
A moderated usability test gives you a wealth of information — some in the user’s facial expressions, others in the movements of their mouse, and still some more in the hesitant tone of their voice when they’re talking out loud.
You’ll want to try to pay attention to all of the above. It does get pretty challenging, which is why it helps to:
“Do you like Feature X?” is a bad question to ask for 2 reasons. First of all, your user’s likely response to a question like that will be “Yes, I like this” or “No, I don’t” because of the close-ended nature of the question. As we covered above, what you really want to know is why! Second, your question isn’t neutral (you’ve embedded the word “like”) and because people often default into responding politely, they’re more likely to say “Yes, sure I do!”
A better way to phrase the same question is, “How do you feel about X feature and why?” You’re not leading the user in any particular direction, and you’re more likely to get lots of amazing insight, beyond just a simple “Yes” or “No”.
The point of a usability test is to see when your users are confused, which they will often be. When they are confused, they’ll default into asking questions — “What does this icon mean?”, “Where’s the right tab?”, and on and on. These questions are actually amazing data points — they point you to where your users are really, truly confused and where their expectations from your product were not met.
What you do not want to do is start answering all these questions during the test itself — that’ll take you off track from the task you’re testing for and may give the user information mid-task that wouldn’t otherwise be available in a natural setting.
Instead what I often do is let users know upfront that I may not answer all their questions (but they should keep them coming anyway), note their questions down and then answer them at the end of the task. That way I can dig into the reason these questions came up at all (more why follow-up questions!).
Caveat: Answer your user’s questions if they are blocking their progress on the task! (e.g., if the user is struggling with a prototype limitation, you can jump in to answer their question)
Usability tests can be one of the biggest challenges of self-control (think Marshmallow test) because your job is to watch your user struggle through your design, but can’t say anything. At All.
You’re trying to see what your user would do if you weren’t around so you have got to simulate that experience. Sadly that means zipping it and silently watching them struggle through your prototypes 😭
Now that the basics are covered, it’s time to get into the actual moderated usability tests interview scripts.
A script is very important for any user research interview, but with a usability test it’s even more crucial since you need to guide users through very specific tasks.
Moderation in a usability test is all about setting the stage and observing your users.
During the introduction and warm up phase of the study, focus on making sure your participant is feeling comfortable and relaxed. If you are conducting the usability test in person, watch out for the participant's body language as well.
Once you've given your task to the user, your job is to observe. Pay attention to where your users look frustrated, pause for a long time, or otherwise show signs of confusion.
Do not interrupt, help your users, or jump in as they talk through the task. It's hard, I know! But if you interrupt your user or start answering their questions in the middle of the task, you are endangering the validity of the test.
In real life your users won't have someone to answer their questions. So you can't answer them during the task either.
Once a task is over, feel free to answer your user's questions before guiding them to their next task.
Many incredible teams like PandaDoc, Growens, 15Five, and Obviousave already created moderated usability testing scripts that their design & research teams are able to leverage every time a new study comes up.
To help you get kicked off faster, these expert teams were gracious enough to let us share their knowledge & resources with you:
As one final parting gift from me to you, I wanted to share a checklist I usually keep handy during usability tests with reminders of the key tasks I need to complete. Pull this out before every call to make sure you haven’t missed anything!
Looppanel automatically records your calls, transcribes them, and centralizes all your research data in one place
