8 Biases UX Researchers Should Be Aware Of

You’ve spent ages clarifying your thinking (maybe atop a mountain even).
You’re still likely to fall prey to a few biases from time to time.

As Soll, Milkman, and Payne mention in their HBR article,

“Even the smartest people exhibit biases in their judgments and choices. It’s foolhardy to think we can overcome them through sheer will. But we can anticipate and outsmart them by nudging ourselves in the right direction when it’s time to make a call.”

As a researcher, you would rather not be caught being foolhardy.

1. Selection Bias: Choosing a convenient participant pool.

Your participants should ideally be selected randomly from your target population.
Selection bias creeps in when your pool of participants isn’t reliable.

Two forms of selection bias:

• Sampling bias: This is when you pick a non-randomized pool of participants. If you pick your flatmates (just because they were in the room with you) to test a website built for elderly homeowners, your research will have a sampling bias.
• Favouritism: This is when you perform multiple studies but pick the ones with results that reflect your opinions. If four of your roommates said that the app was hard to use but two of them found it easy, you shouldn’t just focus on the two who called it easy.
  1. You should also move out of the frat house.
  2. Favouritism is also a form of confirmation bias.

How to overcome Selection bias in UX Research?

• Always select your research participants randomly from your user base (or target user base, no judgement).
• Pick the largest sample size you can manage within the time you have for your research.
• Take all the data you’ve collected into account. Don’t pick and choose.

2. Confirmation Bias: Valuing information that confirms your beliefs

All the inputs you receive during your research should be taken into account.
Confirmation bias pushes you to disregard some of the information that goes against your preconceived notions and opinions. 
You instead end up only listening to the data points that confirm what you already believe in.

If you are running a user interview and the user says something you disagree with, you should make note of it and not enter a debate with them. If you are running a usability test and the participant uses your product in ways you didn’t intend, you should try not to correct them — instead, follow them as they figure your product out.

Debating and correcting participants may be keeping UXRs away from incredible new insights they hadn’t considered before.

How to overcome Confirmation bias in UX Research?

• Use a wide variety of research methods and collate your final results from all of their findings.
• Involve as many stakeholders as possible in the analysis process. Getting multiple perspectives will help cancel out their respective confirmation biases.
• If you are the builder/designer of a product, get someone else to run the research for you. They won’t have the attachment you do.

3. Social Desirability: Saying what’s socially acceptable.

As a social species, we all tend to moderate our conversations to what we consider socially acceptable or favorable for others in the conversation. This is great when it prevents bar fights from happening, but is disastrous for your research.

This bias can creep in for participants in UX research studies and the researcher’s job is to ensure that it doesn’t happen.

Participants may provide overly positive feedback about a product to avoid appearing critical. They may under-report any difficulties or challenges they encountered during a usability test to avoid feeling incompetent. This can result in an inaccurate representation of actual user experiences and behaviors, leading to flawed conclusions and design decisions.

How to overcome Social desirability in UX Research?

• Establish rapport with participants, assure them of confidentiality, and emphasize that honest feedback is valuable.
• Use neutral and non-leading language in questions and prompts to avoid leading participants towards socially desirable responses. Avoid questions that imply a "right" or "desirable" answer and encourage participants to share their true thoughts and experiences.
• Analyze data with a critical lens, considering the possibility of social desirability bias. Look for patterns or inconsistencies in the data that may indicate potential bias and consider alternative interpretations. Mixed methods research will help here as well.

4. Recall Bias: Remembering things way differently than how they happened.

We are terrible at remembering things. Can you recall the 2nd bias we discussed without scrolling up?
Recall bias creeps into UX research when participants remember events in a way that’s disconnected from how they actually happened.

If you are interviewing someone about their experience with a new feature over the past week, it’s highly likely that they aren’t describing their experience accurately. This isn’t because they are evil (most of ‘em), they just don’t remember.

How to overcome Recall bias in UX Research?

• Ask participants to provide specific details about their past experiences or behaviors, such as timelines, locations, or contextual information, to encourage accurate recall and minimize reliance on general or vague recollections.
• Involve as many participants as you can. Multiple accounts will help tone down the impact of recall bias.
• Provide participants with prompt cues or aids, such as screenshots, videos, or reminders of past experiences, to help them accurately recall their past behaviors or experiences during the research study.

5. Hawthorne bias: Behaving differently under observation

This bias is named after a 1920s study that concluded that workers would improve their behavior just because they were being observed. Researchers should be mindful of the Hawthorne effect when trying to study true user behavior.

A word of caution:

As Will Kenton writes in this article, the presence and impact of the Hawthorne effect has come under scrutiny of late.

“modern attempts to replicate the Hawthorne Effect have been inconclusive.
Only seven out of 40 such studies found any evidence of the effect.”

Even though it makes intuitive sense, this lack of evidence should compel you to take the Hawthorne bias with a grain of salt.

How to overcome Hawthorne bias in UX Research?

• Conduct research in naturalistic settings that closely mirror the participants' real-world context.
• Take a backseat as an observer. You can’t observe someone covertly (you want to be ethical), but take a backseat and don’t assert your presence too much.
  Some data collection methods such as heatmapping help reduce the effect as well.
• Provide participants with opportunities to familiarize themselves with the research setup, so that they may answer more normally.

6. Cultural Bias: Cultural differences impacting the research process.

UX researchers bring their cultural perspectives, assumptions, and biases into the research process. So do participants.

This can lead to misinterpretations or misrepresentations of user experiences.

If you are conducting research for a dating app and are from a culture that considers dating to be taboo (South Asia represent!), you might value privacy and secrecy more than your participants from cultures where dating is normalized.

How to overcome Cultural bias in UX Research?

• Conduct pilot testing of research instruments, prompts, or designs with members of the target cultural groups. This will let you identify any potential pitfalls.
• Reflect on your own cultural biases and assumptions that may influence the research process and outcomes.
• Ensure that the participant pool is diverse and includes individuals from different cultural backgrounds, ethnicities, age groups, genders, and other relevant demographic factors.

7. Anchoring Bias: Participants are influenced by the initial information presented to them.

If you are negotiating with a seller over the price of an article (say a statue), the seller might use a trick called anchoring. They’ll name the first price and you will end up negotiating around that price, instead of deciding something independently. You’ll get “anchored” to their initial offer and put too much weightage on it.

This is fun when selling statues, but you should stay away from anchoring bias in UX Research. This bias creeps into a user’s responses if you provide non-neutral information upfront. If you present your product in a positive light from the get-go, the user is likely to moderate their own responses in a way that conforms to your initial information.
This prevents you from getting genuine user feedback.

How to overcome Anchoring bias in UX Research?

• Use neutral language when you present any information about the product. Do not paint a positive or a negative picture of the product — that’s the user’s job.
• Include quantitative data in your results. It is harder (although not impossible) to let anchoring interfere with survey results, user interactions, and weekly active users for example.

8. Fundamental Attribution Error: Blaming the user and not the product

We are more likely to attribute others’ mistakes to their faults and not to situational factors. This comes up in UXR as a tendency to blame the users for their inability to interact with the product, instead of the product itself.

This impacts both researchers and users, both of whom end up blaming the user. Was it strange that the website could only be navigated using a joystick? No — it’s the user’s fault for not having one at their office.

How to overcome Fundamental Attribution Error in UX Research?

• Attribute all UX flaws to bad design/copy as a rule of thumb. The only valid reason to attribute it to a user is if no other participants run into the issue.

• Even then, consider why the one participant may have this issue. Are they from a unique cultural background? Do they have any accessibility needs that your product doesn’t provide?

These biases scratch the surface of how irrational our decision-making often is.
The greatest antidote to this irrationality is consistent research with a wide variety of participants.

Looppanel makes user interviews so effortless, you can’t help but run them consistently.
Here’s an action step against bias: Sign up today!