Do we look at some users worse than none?

Abstract: Seeing is believing, observation is a way to study the very few users interact with the product, rather than to look at reports or presentations that are professional. However, if in a usability test, the investigator only observes the time of one or two users,

Seeing is believing, observation is a way of studying the very few interactions between users and products, and more persuasive than looking at professional reports or reporting.

But what if, in a usability test, a researcher observes only one or two users ' time? In what circumstances do we observe that some users do not observe worse than one?

Watching only one is worse than not observing

Imagine if a researcher, such as a product developer or designer, they found 5 users to test, and these users did not feel any confusion or difficulty in using the product, and they mistakenly concluded from the 5 users that all users had no problem using the product. And will think that the product has been very good.

Two user mode

To avoid this one-sided result of a single user survey, some research teams have developed simple minority compliance rules that ask, or in some cases, to observe the use stages of multiple users. If the researcher can only observe once, then this is much worse than observing nothing.

If all usability tests are done according to plan and the results of their efforts are consistent with the distortions, then you will understand why there is such a rule. With at least two users, you'll see how different users interact with the software, and how they can be more fully summarized.

Three users break the draw

Another form I've seen is to observe at least three users, not two. A third user could break the situation when two users could lead to a draw and not make a decision. At the very least, it would be appropriate to avoid preconceived ideas affecting the final conclusion.

Why one is better than none

I sympathize with the researchers, who are often led to distort the data with preconceived notions. So it is not difficult to understand such guidelines. However, I do not want to say that watching one or two users is worse than doing nothing. Because it involves a probability problem.

The table in the following figure, which represents 5 users in the usability test, respectively. White squares represent users with similar experiences, and red squares represent a user who has had an unusual experience with a product (unusually good or bad).

The researchers will see that the probability of this unusual user is one-fifth or 20%. Therefore, in any study of only an unusual user, there is a possibility of being misled.

However, the law of large numbers tells us that, over time, it is more likely that the researchers will see a typical problem better than an atypical one. Of the following five studies, each group had five users. Each group of studies has a white square that represents a different red square and four consistent experiences.

Using the two-item probability formula, one researcher will see that the probability of an unusual user in all five observations is 0.03%. The probability of using a researcher to look at three unusual experiences is 6%. The more you observe over time, the more likely you are to see typical problems.

Problems occur

We extend the discussion to the problems that users are experiencing. Similarly, using the rules of probability, allowing 5 of users to uncover the most obvious problems also means that only one user-observed problem is more likely to be apparent than those with unusual problems. That is, if you look at one-fifth of users who have this problem, the problem can be 8.5 times times more than just 20% of the users who are testing in 1% of the users.

The only problem

However, when it comes to problems, any user who is being tested will encounter many problems, and depending on the type of research, many of these problems can only be seen once-although you have tested many users. For example, the following grid shows a problem matrix for a usability test with 30 users involved. A total of 28 problems were recorded. Of these, the 9th question (32%) was encountered by only one user.

One obvious disadvantage of observing only one user is that stakeholders cannot distinguish between unique and more common problems. Only one user, the conclusion will be that these issues affect all users. With two or three users to test, at least we can distinguish between these unique and redundant issues. However, we cannot overestimate the recognition we receive from the one or two or even three of users.

The uncertainty of a small sample

The key to getting a third person to win is a compelling idea (known to every school child), and in fact, it doesn't necessarily require a lot of users. For example, if one-second of users have a bad experience, 90% we can assume that 12% to 88% of all users are experiencing this bad experience. This range is 76% accurate. By adding a user to break the balance of the situation, we can have 90% confidence that this problem will affect all users of 25% to 93% (68% precision). So when we look at the user from 2 to 3, we've increased the 800 point precision. Although only 8%, two ranges are still huge.

Conclusion

When I say that observing two or three users is better than observing one or 0, it does not mean that observing two or one is worse than not observing one. Observing a user means that we have no way to assess the many different ways in which users interact with the product. It is important to observe more users, and it is often equally important to identify and resolve usability issues.

Just observing a single user also tests some of the problems that affect the interface design. Over time, if researchers look at a random user in each usability study, they will focus more on the unusual issues than the common ones. In any given sample study, they always have a good chance of being misled by an unusual experience, and over time, even a separate sampling will focus them on the most common problems.

If a researcher's understanding of the data is bad, I think more research will only lead to more misinterpretation of the user. Therefore, it is no good to believe that most people are better than few and I still believe that some user research is better than none.