How to recruit for and run remote user testing

When it comes to running user research and testing, teams can often be strapped for resources. A lack of time, people, budget or participants can make completing valuable research a challenge.

But with remote user testing, you can conduct all of your tree testing, card sorting, first-click and usability testing online — giving you results quickly and easily.

What is remote user testing?

Picture a window that allows you to peek inside the lives of your users, a way of observing or getting feedback from them without majorly disrupting their routine and environment. There are many ways to describe remote user testing, but this is a favorite.

Unlike traditional usability testing conducted in a viewing facility or lab, remote usability testing does not require hiring a facility or having to set up testing sessions in a physical location. Sometimes, it doesn’t even require a moderator or person leading the session, depending on the type of remote session you plan to run.

Remote user testing, using video call software, online testing tools, screen-sharing tools or just a simple phone, allows you to conduct research and usability sessions without asking participants to considerably change their everyday routine. This is a much more natural testing environment than a research or testing lab — how often are we shut in a room to concentrate on one task for an hour without interruptions?

When to do remote testing in the design process

Let’s consider a “normal” user experience project with five or six stages (user research, building personas, wireframing and prototyping, etc.) and a few sprints in the middle. With this scenario in mind, there are two optimal stages to conduct remote user testing, especially if you’re thinking about running unmoderated sessions.

If you are running Treejack sessions, for example, unmoderated remote testing is a good place to start. For this to be successful, you will need to carefully craft the questions or guidelines that will guide your user during their session in Treejack. The time will come when you will need to find out more about their thoughts or actions when testing a product, and this is where remote moderated user testing comes in.

Remote usability testing – both moderated and unmoderated – can also be useful further down the line when you progress into a project, but need a quick turn around when trying to fine tune the details and test quick changes.

What are the benefits of remote user testing? And what about the cons?

Remote usability testing, regardless of the type, can be the perfect complement to face-to-face usability testing. If running sprints or trying to get the big picture when going into a re-design project, for instance, it might actually be all you really need in order to identify the key focus.

Even though user and researcher are still in different locations, it’s possible to briefly examine facial expressions and body language. These can become real indicators of how easy or complex a task is! Moderating a session makes it easier to ask the user to develop on a certain comment they might have made or provides the opportunity to reword a question that might have been misinterpreted.

Another supposed advantage is that it’s easier to recruit people to take part in remote user testing. Although this is partially true, we recently received some very interesting feedback from our community of users that proves this is not always the case. A study we recently worked on with the research team People Thinking revealed that 48% of the participants interviewed would not be interested in taking part in remote sessions using Skype due to concerns about their technical ability, that a video call could feel intrusive or that they wouldn’t be able express themselves well via this channel.

So, although remote moderated testing is likely to help you gain valuable insights, understanding the user’s environment and personality without asking them to leave their house or work, there is potentially a downside to it. Exclusively running remote user testing or research could alienate nearly half of your users; therefore, a mix of different types of research and testing seems to be the best option for any big project.

Of course, we understand this is not always possible. Remote testing, especially unmoderated, is more financially viable if you require large numbers of participants or regular sessions in a relatively short period of time. They will still produce valuable insights and are a perfect option for specific stages of the design process.

The bottom line is: don’t try to find excuses like lack of money or time to avoid testing your idea or product. Remote user testing is definitely doable on a budget and in just a few hours. Run some testing, rather than no testing at all, and the benefits will be obvious in the long term.

How to recruit for remote user testing

Why you need to look for the right kinds of participants

Although the idea of rounding up five or six people for research might sound easy, this is not always a streamlined process. Even in the case of remote testing, it is important to find users who aren’t “serial testers”, whose demographics and behaviors suggest they are likely to use your service or product, and/or people who have a specific device or phone. The bottom line is: you still need to search for the right users and this might not be the best use of your time or skills.

This is where external participant recruiters, like People for Research, come in handy. If you have some budget to accommodate recruitment costs and incentives for participants, then an external recruiter will have the expertise and network to take this task off you.

All recruiters offer slightly different services, some more comprehensive than others. People for Research, for instance, will help with some tasks besides recruitment: this includes helping to define your recruitment brief, writing the screener, setting up an online questionnaire to determine suitability and then screening all participants over the phone to confirm they are perfect for your project, while also determining how talkative and comfortable they are when sharing personal details with strangers.

At People for Research, we find that building a relationship with our community before recruiting them to take part in research really helps during crunch time. It is paramount to keep your pool of potential participants engaged and build trust in your brand; hence why it could potentially be a lot more difficult for UX designers and researchers to try to recruit users by themselves.

Users don’t know you, and most people are new to the concept of getting paid for their feedback. So even getting bank details so you can pay the participants may be challenging, as you have not had a chance to build a trusting relationship with them. Overcoming these barriers is likely to delay your project and, even if you manage to find people to take part in your remote user testing sessions, there’s always a possibility that they might not be the right people.

How to screen your participants

Screening is the part of the recruitment process where you check the participants’ suitability by going through some key questions. For example, you might only need participants who have purchased a car in the last year. A screening question can help you screen out the wrong participants, so you can get more targeted data. It is also the stage where you should get informed consent from the participants and give them a chance to ask any questions they may have about the research, such as what you are planning to do with any data you capture. Again, this is something a specialist recruiter will be able to help with.

If your budget won’t stretch to a recruiter, then we recommend making time to write your screener when you are defining the users that you hope to recruit. Think about what questions you will use to check suitability, and how you can incorporate these in a clear way that is not going to be leading the participants. Be clear: for example, don’t ask participants if they often order food using an app, as the concept of ‘often’ changes from person to person. Always test your screener on someone unconnected to the project so you the chance to check it makes sense to others.

When putting your screener together make sure you include one or two open questions, even if you don’t need them to meet your recruitment criteria. This not only allows you to start to get to know the participant a little, but it also allows you to assess their communication skills and whether they are willing to share more than the most basic answer. This will avoid you having a situation where you feel like you are pulling teeth to get anything useful out of the participant during the session.

Considering Jakob Nielsen’s ‘think aloud’ strategy, where participants are given representative tasks and asked to continuously think aloud while using a system. It is important to try and assess while screening participants that they will be capable of this process, as it does not come naturally to all participants. Your ability to assess a users non-verbal communication, such as body language, is diminished during a remote session so you will be more reliant on what the users says aloud. Therefore, we always recommend having at least one open question in each screener.

Remote user testing is an extremely valuable addition to any project, as long as you have the right users by your side. And, even if you don’t have enough budget to use recruitment services or to pay each participant, remember that a little bit of testing is better than none!