Many UX teams rely on remote usability testing to efficiently get design feedback from users. There are two types of remote user testing:

• Moderated remote testing involves a  researcher meeting with a participant via remote screen-sharing software , which allows the researcher to provide instructions, observe the user’s interaction with the design in real time, and ask followup questions specific to that participant’s session.
• Unmoderated remote testing does not require a researcher to attend each test session; instead,  a software application provides instructions to users, records their actions , and may ask them predetermined followup questions.

Is Unmoderated Testing Right for Your Project?

Unmoderated studies do not include any direct interaction between the researcher and the study participants, which is both their biggest benefit and their greatest drawback.

Because there’s no need to schedule an individual meeting with each participant, unmoderated testing is usually much faster than a moderated study.  It may be possible to launch a study and receive results within just a few hours . Unmoderated studies also allow you to collect feedback from dozens or even hundreds of users simultaneously. And for international studies, you don’t have to get up at an ungodly hour to match users’ time zone.

However, there are important limitations of unmoderated usability testing:

• Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype.
• Without a moderator, participants tend to be less engaged and behave less realistically in tasks that depend on imagination, decision making, or emotional responses.

To better understand this second limitation, think, for example, about the difference between shopping and purchasing.

Shopping can include many different types of research and comparison — there’s no single ‘right’ way to do it. In order to shop realistically, participants must first imagine themselves needing that product, then pay attention to details and make comparisons. A participant who just pretends to shop and isn’t very motivated often will glance only at a few products and quickly select one that seems reasonable. But, in real life, consumers who are spending their own money on a product they actually need behave very differently.

Of course, a moderated study is not totally realistic either, but because participants are aware that the moderator is observing them, they will be socially motivated to fully engage with the task. Thus, this social pressure compensates for the lack of personal motivation in moderated studies — and even more so in in-person studies.

  Unmoderated studies work best for evaluating live websites and apps or highly functional prototypes . They are appropriate for studying activities that don’t require a lot of imagination or emotion from participants.

Unmoderated research requires even more meticulous planning than a moderated study, since you can’t rely on human judgment to adapt the study procedures on the fly.

For an unmoderated usability study, you’ll need to go through all the steps below:

1. Define Study Goals and Participant-Recruitment Criteria

Choosing software should not be the first step in unmoderated research. Before you decide which testing software to use, you should get a clear idea of what you hope to accomplish by doing the study. Clearly articulated study goals allow you to identify must-have requirements for the testing software.

If the study goal is to…	The study tool must be able to:
Compare how long it takes people to complete a signup and checkout process on your site vs. your competitors	Measure time on task Aggregate time on task data Display individual times to inspect outliers Export time data to a standard spreadsheet format (e.g., .cvs)
Help a large team understand why users struggle to complete the checkout process within a mobile application	Record screen and audio of native mobile applications High-quality video capture, to see detailed interactions Easily create clips and compilations of videos to share with the team

Goals vary from study to study, and a tool that is well suited to one study might be not at all effective for another. Knowing the study objectives is essential in order to make a good tool choice.

In this stage, you will also need to think about the types of participants you want to include in your study. What should their demographic be? Where do they need to be located? Will they be users new to your system or experts? Will they match a particular persona or user group in your target audience? These questions usually are determined by your study objectives and can also inform your tool selection.

2. Select Testing Software

For unmoderated studies, the software that administers the test is absolutely crucial to getting useful results. The software must guide the participants through the session and record what happens. It may also control the selection of study participants.

Fortunately, there are many different unmoderated testing services available, with varying combinations of functionality at different price points. The plethora of choices means that you can now be more critical in selecting a tool that suits the requirements of your project. (Features and pricing change frequently for many of the unmoderated testing tools, so be sure to compare them to your needs at the time of your study.)

These are only some of the available tools. There are more to choose from.

3. Write Task Instructions and Followup Questions

Many unmoderated testing services include study templates with generic example tasks. Don’t blindly copy them. The tasks that you give participants to do on your site or application should be highly specific to your situation.  Generic tasks, such as “What is the purpose of this site,” are unlikely to give you good insights : to really assess the usability of your system you will need to write your own tasks.

  Writing tasks is where most researchers fail  in getting the results they need from their studies.

To write good task instructions for an unmoderated study, first articulate what it is you want the user to accomplish (such as: use the help section to answer a question, upgrade an account, or save an article to read later).

Then, describe that objective with instructions that are specific, realistic, and actionable — without including hints that make the task too easy. You’ll need different types of task instructions depending on whether you’re doing a qualitative or quantitative study.

In unmoderated studies, the activities that you want the participants to conduct have to be even more carefully written than the tasks for moderated sessions. Participants cannot ask for clarification if they don’t understand the instructions and you can’t ask them to try again if they do the wrong thing.  If users misinterpret your instructions and perform the wrong task, your test is wasted . Unmoderated task instructions should also explicitly tell users when they should stop; remember, the moderator won’t be there to ask them to move to a different task.

You should also meticulously plan any followup questions. These can include quantitative questions, in which participants rate the subjective difficulty or satisfaction of an activity. Or you may ask open-ended questions which prompt users to describe specific parts of the experience. Carefully choose how you phrase your questions; broad wording such as “How would you describe this brand?” may lead unmoderated participants to talk about their past experiences instead of the system they just used.

4. Recruit Participants

There’s no point in watching people use your system if they don’t match your target audience. Make sure you have some control over who participates in the test, either through screening questions, or by recruiting your own participants.

Some tools only offer unmoderated user testing with participants from their panel, while others will provide you with a URL to distribute to your own pool of testers.

Using a provided participant panel is fast and easy (especially if your product is something that is relevant to a broad consumer audience). Panel participants are familiar with the study software, too, and can participate in your test as soon as they have time. Almost every panel includes some basic demographic filtering, but  most studies get better results if you screen participants using questions about their behavior, not just about their age or gender .

If the experience you’re testing is relevant only to people who meet specific criteria such as driving a car or shopping online regularly, make sure to use a tool which allows you to write your own custom-screening questions.

The downside of using panel participants is that many do these studies so frequently that they’ve learned to focus on certain aspects of the design and look for things to critique. To compensate for possible “professional testers,” recruit extra participants and exclude data from people who didn’t seem honestly engaged with the activities.

Being online means that you can test users on the other side of the globe as easily as people on the other side of the street. Many unmoderated testing services now have panels which include participants from all around the world. If your site targets international customers, unmoderated testing is a great way to reach a wide variety of locations.

In summary:

• B2B sites, sites that target elite or rich customers, and other services with narrow target audiences usually can’t use panels and must recruit the test participants themselves.
• B2C sites and other services that target a broad audience usually get faster and cheaper results by using a panel.
• You don’t always need the biggest panel, but if you require users from a particular region some panels may take a very long time to fulfill your study. (If you really need a specific audience, discuss your requirements with the testing service or panel manager in advance, and if they’re at all vague about whether they can do it, consider using a different service.)

5. Analyze Results

Unmoderated studies can quickly accumulate a LOT of data, so you’ll need an organized, analytical approach to turn this data into actionable insights about your design.

If you collect qualitative data, such as video and audio recordings of participant actions and comments, you’ll need to review each session recording. Users’ verbal and written comments can be misleading, so you have to watch their behavior in order to understand what works or doesn’t. In a moderated study you can follow along with the participants as they conduct activities, but during unmoderated studies you need to be able to watch a recording afterwards.

  In each recording you’ll want to identify problems, questions, and both positive and negative reactions to the design . This process can be relatively quick if you have only a few recordings to review, but for large studies with dozens of participants, video analysis becomes extremely time-consuming. If you expect to analyze large studies (or to carry out several different smaller studies), look for an unmoderated testing tool which offers robust features video analysis, particularly:

• Tagging videos with timestamped notes as you watch them
• Aggregating, exporting, sharing, and visualizing the notes you’ve added to your recordings
• Producing short clips or highlight compilations of important moments in your recordings

For extremely large qualitative studies, consider tools which can collect some quantitative measures or which offer automatic transcription. These features don’t eliminate the need to carefully review your recordings, but they can certainly expedite the process by directing your attention to specific recordings that are likely to be significant (such as recordings where users had low satisfaction ratings or recordings where particular keywords were mentioned).

If your study is primarily quantitative, your analysis will be quite different. Metrics such as success rate, task time, and subjective ratings will be automatically collected by your study tool. But to ensure your conclusions are accurate you’ll need to review your data.

Conclusion

Unmoderated research requires less work than moderated testing during the session, but it requires meticulous advance planning before the study begins.

You also can scale it better, while still retaining the valuable feedback and qualitative stuff too. You just need to be well-prepared and do half the work beforehand.

Unmoderated User Tests: How and Why to Do Them

PS. User testing (moderated or not) is one of the most valuable activities you can possibly do. It is also one that is criminally underutilized. Insights uncovered during user testing are invaluable and you always find surprising things.