Welcome to Happy Stories! Every week we’ll keep you up-to-date on the latest developments in the area of UX research and testing of digital products. Tests ensure happy users and that makes us happy. Hence the name; Happy Stories. The godfather of usability testing, Jakob Nielsen, conducted an elaborate research together with Tom Landauer to find out what the magic number of respondents for user testing.
At the end of this article you will know:
✓ The number of respondents you’ll need for your tests
In short; fifteen is the number of respondents to find all usability problems in your design. Nielsen and Landauer created a formula to determine the number of usability problems:
N (1-(1- L ) n )
N is the total number of usability problems in the design and L is the proportion of usability problems discovered while testing with one user. They found out that the average value of L is 31% throughout a large number of projects they studied. If you plot the curve for L = 31% you get the following result:
The graph teaches us a number of things:
- Testing with zero users gives zero insights, so testing with one single user is always better than not testing at all
- Testing with a single user already give you about 30% of all usability problems in your design
- Testing with an additional user shows that there is some overlap but in findings, there will also be some new insights.
- As you ad more users you’ll get less and less new findings because you’ll keep seeing the same problems again and again. After fifteen respondents you generally won’t find any more problems
Five is the magic number
Although the graph shows that you have to perform 15 tests to get all the problems out of your design, but this is not recommended. The main reason for this is budget. You can better distribute your budget over a number of small tests than make up everything for one big super comprehensive test. You better perform three tests with five users than one test with fifteen users. Ultimately, you want to improve your design and not only report the pain points. After the first test you can already solve 85% of the pain points in a redesign. At least, that’s the goal. But a new test will have to show whether the problems have actually been solved.
When to test 5+ users
There is one big comment on all of the above and that is the target group. In the above model, we assume that you are testing one uniform group of very similar users. For example, if you have a website or app that is used by different target groups, it is important that you also test with five people from the different target groups.
The biggest takeaway of this whole story is that no matter how many respondents you test, it will always benefit the quality of your product. But if you do it anyway, do it well! Five seems to be the magic number when it comes to testing a specific group, but you test more target groups, then perform multiple tests. Do you have any questions about usability testing or finding suitable participants for your tests? Please contact us.