Why do Web projects require only 5 people to test?

People always think that user testing is a complex and costly thing. A website design project requires a large budget and a long timetable. In fact, usability testing is not something that most people think is a waste of resources. Within the range of your affordable test costs, you can achieve a good test result with no more than 5 users.

In the course of the initial study, Tom Landauer and I came up with a formula for usability testing, about the number of testers:

Suppose the number of testers for a usability test is n,n the total number of problems found for all usability tests, L is the problem discovery rate for a single tester.

N (1-l)ⁿ)

We found that the average problem discovery rate for most projects was 31%. When l=31%, you can get the following:

When the user is zero, the number of problems found is "0 user 0 discovery." We can see that the first user almost found One-third of the design usability problems, and 0 found quite different. When we look at the second user's test case, we can see that some of the problems are coincident with the first user. There is a certain difference in behavior or discovery between people, and the second user can find something different from the first user, a new discovery that distinguishes it from the first, but the number of problems is less than the initial discovery of the first user. The third user did a lot of things that were repeated before, and some even repeated them 2 times. Also, of course, the third user's own ability found a small number of new problems, apparently a lot less than the first two users.

Now, with more testers (user), you'll find a phenomenon that repeats itself as the number of people increases, and the discovery rate of new problems is starting to drop. Obviously no longer need to let more people repeat the problem of finding repetition, naturally back to redesign to solve the problem of discovery.

By the 5th user, you're wasting your time doing repetitive things and almost no first discoveries.

Iterative Design

The curves in the diagram above are clearly indicated, requiring at least 15 users to discover all the usability problems in the design. But why am I more inclined to recommend less testers? One of the main reasons is to reasonably allocate the budget for usability testing. Let's see, when you're recruiting 15 client representatives to test your design, it's 3 times times the cost of spending 5 users!

When we do user usability testing, the ultimate goal is to solve or improve the actual design, rather than just get a written report. When the first test of 5 users found 85% usability problems, you could fix them in the next desagn.

If you want to find more problems, of course you need to test again. Even if I say redesign can fix the problem that was discovered during the first Test, the fact is that you may think that the new design can be used to support existing problems. However, until no one can design a more perfect user interface, there is no guarantee that the new design will fix the problem. The second test can return to existing problems and confirm that they are fixed. At the same time, a new design means that a new user test is needed.

The second test for 5 users reveals 15% of the problems left over from the first Test. (There will still be 2% of questions left to wait until the third Test is discovered).

In the end, the depth of the second Test can be designed into the design architecture of the Web site to get some of the information we need, such as architecture, task flow, etc. to meet customer needs. These problems are often overlooked after some superficial usability problems are hidden.

So the second test was the quality assurance of the first Test, and more in-depth questions could be found. The second Test will provide a new list of issues for the redesign of the system, but the number will obviously be less than the first Test. However, this test does not fully complement the first Test, but also requires a third test to check the leak fill.

The final Test effect of 5 users three times is much higher than the effect of 15 user tests at a time.

Why not suggest a single user's test? A single person's behavior always has some risks, and there are always some unexpected, uncertain behaviors.

You might think that 15 users would do a round test, which would be better than 5 users doing 3 rounds. Because the curve shows that the first user's discovery rate is significantly higher than the subsequent users, but why do we insist on multiple rounds of testing? Two reasons: First, by looking at 3 people can see the diversity of user behavior, can be seen that the behavior is unique to those who can be generalized; second, the cost-benefit analysis of user testing shows that the optimal ratio of 3-5 test users depends on the style of the test. The initial cost of running is related to the test plan, and multiple users are better at reducing the initial cost effect.

when to add more test users

When your site customer base is covered by several different heights of users, you need to add more test users. The formula above applies to Web sites where the user base is close to or varies by an hour. For example, when your site client base is positioned for your parents and children, you need to take full account of the behavior of two different groups of users. The similarities between the systems are the connections to the sales staff of the purchasing agents.

Even when the user community is completely different, there are many similarities between the two groups of people. After all, all the users are people. All usability issues respond to the interaction and impact of human behavior with the site.

When testing a different user base, you don't need a lot of testers (users) in each user base, and one of the following testers is enough: 1, if you test two user groups, each group needs 3-4 users; 2, if you test 3 or more user groups, Then there are 3 test users per class (at least 3 users will be able to ensure that you are overwriting different behaviors within a user base)