Conversations lead to actions, reports lead to filing cabinets

There are three types of conversations that occur during a usability study: in your head, with the participant, and with the team. If you remain engaged during each of these conversations, your words will have more effect and your job will be much easier.

During a usability study I’m focusing most of my attention on my participant, but I also pay attention to the voices in my head. They are always checking in to make sure we’re still on track to answer the research goals we set as a team ahead of time.

If I just keep watching and listening to my participant completing their task, will that help the team with their immediate issues? Will it help them with longer-term design input?

If instead I interrupt and change the course of the task, will that just pull my participant out of their flow, or will it lead to better insights? Are they quiet because they are deep in thought, or because they are hopelessly lost?

That’s the second kind of conversation happening during the study; between my participant and me.

In summative studies, this conversation is mainly in the form of a monolog from my participant, with my only contribution being to say “Please remember to think out loud” if their monolog tails off. Anything else I say is likely to change the outcome of the study, and if I’m tracking metrics or checking for understanding, this would count as undue influence.

However, in formative studies sometimes it’s fun to dig deeper on some issues. I might take the time between tasks to ask a clarifying question, by first listening to the participant, then probing on a certain thing they said or did, and finally validating my understanding of their response by summarizing it back to them.

At the end of a session, I might take participants back to a part of the interface where their behaviors raised questions from the voices in my head. After steering them back to that part of the interface, I’ll again probe and validate in order to help answer broader research questions that the team may have agreed upon.

Immediately after the study my conversation moves to the team. Did they see the same things as me? If not, what was it about the perspective that they or I brought in to the study that made us focus on different aspects of user behavior and interpret things the way we did?

It’s good to clear this up before they go off and change stuff. Those changes will start just as soon as they get back to their desks so I know I have to have this conversation while I still have their full attention.

I try to focus this conversation on the interface and on what our participants said and did. Only after we’ve got through that information will I talk with the team about changes and solutions.

I normally ask them to write each observation they made on an individual sticky note. Then we put all of those on the wall underneath either a happy face or a frowny face. We’ll agree the biggest issues and successes, which determines the priority of the changes they need to make, and the things they need to be careful not to break in the process. It’s hard for one team member to argue for a pet feature change if everyone else saw different behavior or thinks it’s low priority.

After this, we move the conversation forward to how they plan on making the necessary changes.

I know that UX people often want to create design solutions on their own and then feed them back to the team, but the whole team is ultimately responsible for the success or failure of the product, and the whole team bring multiple insights from business, development, implementation and marketing perspectives. For that reason, I like to respect their suggestions and only redirect them with a little teaching moment if I see interaction design issues with their proposals. This way, they feel full ownership of the changes, and so those changes will happen much faster.

To help move this type of conversation forward, I’ll often print off large copies of each screen. We place sticky notes over areas of the interface to make edits, creating an immediate paper prototype so that everyone understands the impact of the changes that are being proposed.

Creating this prototype means that the changes are defined to a level that they can be easily coded. Field label text is updated, the order of fields or even the flow of whole screens is changed, and the whole team hangs around to participate because they know they’ll miss out if they don’t.

The team won’t wait for a written report to make changes – and why should they? Reports are only really useful as an archived summary of what took place. They should list the session goals, participant actions, and team’s decisions as a result of what they saw. Those decisions are made as a group in the minutes after a study, not on my computer days later. I’ve learned to embrace that fact, not to fight it.

The team conversation writes your report for you, but it is the actions from this conversation that count, not the filing cabinet the report is stored in.