Cognitive biases in user research

6 min read Guest Blogger

Principal User Experience Consultant at PwC’s Experience Centre, Ruth Ellison, shares her thoughts on cognitive biases before her UX New Zealand 2016 presentation.

It was a number of years ago and I was sitting next to this research participant, who was testing our new web tool before it launched. She was telling me a story about having an idea for her business, and the attempt to set it up. As she was telling her her story, she was pulling up a range of screens and flicking between them to illustrate her story. It was fascinating. As she pulled up our tool, I watched with interest and she started navigating the tool. At one point, I leaned in closer to her screen as we knew that this was a trouble spot in our tool. She saw me lean in, and stopped to ask if I wanted her to discuss this screen.  This is an example of the Observer-Expectancy effect, and it’s a cognitive bias. It is a phenomenon that can occur when your beliefs or expectations cause you to unconsciously influence the research participant’s behavior. Think about the subconscious “uh huh”, head nods and smiling when the participant is going down the “right path”. Or scribbling down notes and having the participants say “Have I done something wrong?”.

What is a cognitive bias?

Cognitive biases are psychological tendencies that cause the human brain to draw incorrect conclusions. We take these types of mental shortcuts, because our brains use a number of simplifying strategies and rules of thumbs (known as heuristics) to help ease the mental processing when we’re making decisions. While these mental shortcuts are really useful in helping us to deal with a world filled with complexity and ambiguity, cognitive biases can lead to faulty judgements and decisions.

These cognitive biases can affect our design research in multiple ways. We all want our research to provide reliable input into our projects and most of us wouldn’t deliberately distort data. Yet, we’re human, and we’re all susceptible to many cognitive biases that can affect the outcomes at any stage of our projects. Bias is unavoidable, but being a good researcher is about understanding our inherent biases and how we can minimize the effects. Distorted or misleading results can be very detrimental to a project. It can misinform the direction of a project, or provide false confidence about decisions.

Cognitive biases in design research

There are a large number of cognitive biases that can affect our research. Here are a few examples.

Peak-end rule

The peak-end rule refers to the cognitive bias where people judge an experience largely based on how they felt at the peak and end of the experience, rather than on the total or an average of every moment of the experience. This can affect a participant’s response when you’re asking them to describe the experience that they had while interacting with your product or service.

I saw this happen a number of times across many research projects. On one project, we explored people’s experiences of traveling overseas. One participant told me about her holiday traveling through Indonesia and doing homestays where she got to experience the local culture. She described a number of interesting cultural ceremonies that she observed and noted that she didn’t have any negative experiences along the trip. As we explored her experience in more detail, we uncovered a range of pain points in the lead up to the travel, such as long uncomfortable bus trips, and getting sick at one point. But her recollection of the holiday as a whole was quite positive despite the number of issues along the way.

This peak-end rule has been demonstrated through a number of interesting research papers such as “The experienced utility of queuing: real-time affect and retrospective evaluations of simulated queues” and “When more pain is preferred to less: Adding a better end”.

Primacy effect, recency effect & serial position effect

In a 1946 study by S. Asch, one of the experiments involving giving two groups of participants one of the following sets of characteristics:

  • Group A intelligent—industrious—impulsive—critical— stubborn—envious
  • Group B envious—stubborn—critical—impulsive—industrious—intelligent

Participants were asked to create characterizations of people based on these personality traits. These two sentences contained the same information, but the first sentence placed the positive trait at the beginning, while the second sentence placed a negative trait at the beginning. The study found that the research participants from Group A evaluated a person in a positive light while participants from Group B evaluated a person in a negative light.

This is an example of the primacy effect where items at the beginning of a sequence are easier to recall, which can then affect decision making. In turn, this will affect our research, as the order in which we present information to participants will play a part in the way they respond to our designs or concepts. Other related cognitive biases include the serial position effect and the recency effect, where it refers to the tendency of a person to more easily recall the first and last items in a sequence.

Confirmation bias

Confirmation bias is one of the most dangerous biases. This is the tendency to search for or interpret information in a way that confirms your beliefs. This affects the running and gathering of user research data, as well as the interpretation and reporting of findings.

For example, if you believe that a particular feature in your product is going to be useful for your users, then it’s easy to remember instances where participants liked that feature when you’re running a series of usability tests. But times where participants struggled with using a feature may be accidentally overlooked.

When it comes to interpreting the research results, factors such as internal politics, personal goals or simply lack of knowledge can turn into a cherry picking exercise, where we may consider some results and ignore others. This then affects the narrative that we share with our stakeholders, which in turn can affect the outcomes of our project.

Dealing with these biases

Cognitive biases are part of what makes us human. While we cannot remove our biases, there are a number of ways of dealing with our biases to help us become better design researchers. These include:

  • Triangulate your research methods by using a mix of observational research methods (like contextual inquiries) with other techniques.
  • Consider the order of questions and designs carefully.
  • Use open ended questions and be careful with the wording of each question.
  • Alternate the order in which participants are shown concept or design versions.
  • To overcome confirmation bias, give someone the devil’s advocate role to question assumptions or list assumptions before you start your research. Aim to disprove your hypothesis, not just prove that you’re right.
  • Consider evidence equally – not just the parts that confirm your belief/assumption.
  • Use collaborative analysis sessions as it provides different perspectives and helps to challenge our biases.
  • Strive for objectivity and always assess your method, your analysis and yourself for bias.

Further reading

Interested in learning more? Check out some of these resources as a starter:

Want to hear more? Come to UX New Zealand!

If you’d like to hear more about what Ruth has to say about biases, plus a bunch of other cool UX-related talks, head along to UX New Zealand 2016 hosted by Optimal Workshop. The conference runs from 12-14 October, 2016, including a day of fantastic workshops, and you can get your tickets here. Got a question you’d like to ask Ruth before the conference? You can Tweet her here: @ruthellison

Published on October 3, 2016

Guest Blogger