I like this article a lot. It’s well written and full of plausible and reasoned data.
In my experience you only need to test until you hear redundant feedback to find the most common issues. Sometimes 5 is too many. It isn’t that testing more is a negative (as the article mentions the distribution of issues across tests wasn’t always the same even if 85% of issues were found in first 5).
One of the hardest parts about user research IS pipeline - maintaining a viable stream of testers without greedily overburdening them. I don’t feel technology has made that easier. That’s based on my opinion of testing w random recruits. That’s prob ok for mass appeal apps like a chat or note taking app like Notion (maybe?). But I have used a lot of testing software and found there’s always a gap between general testing v. testing direct testing w your actual users. Real users always reward you w more specific feedback. And if they’re your users you can also collect light ethnographic research w them too (show me how you’re doing that [task] today...).
A platform like Maze or a competing service like Usertesting.com would of course push for more testing. Not necessarily wrong, but I think it misses the pipeline dilemma most researchers are always working on: not burning out leads. There are only so many users to go around, especially so in hybrid markets like healthcare or education.
If I have 25 highly highly relevant users to test w. I’m going to do 5 tests of 5 because I know I’ll get more return. Maybe I even had a prototype fail on invision on the first 2 tests? After the first cohort of 5 I’ll adjust my product and see how the results change in the 3rd set, etc. It’s agile. Now add in that I also have 6 other designers, 10 PMs, 3 user researchers, and a team of product marketers all looking to learn from our users... that’s a huge burden on pipeline.
Anyway I highly recommend reading the article. The idea that you don’t need to always limit yourself to 5 testers is a good idea to know about and accurate. But I’m a bit skeptical 20 is the defacto new 5. That’s prob a step too far.
Thanks for your feedback! You make some very good points. In the end, the number of testers you need will depend on many factors. As I wrote in the article, sometimes five will be enough, sometimes not. It really depends on your project and other factors I outlined.
My goal with this article was to break the cycle of 'five users are enough' and hopefully make people question this assumption before they test. Again, five users could be enough, but it's really important to take this on a case-by-case basis.
And I agree with your point that testing with your actual users is better--that's what we found as well at Maze. However, when access to your own users isn't possible (perhaps you don't have them yet, or maybe it's a sensitive project), then recruiting users is a second best option.
As a final note, at Maze, we advocate twenty users because we do quantitative usability testing, and twenty users would allow you to get valid results. We outlined why in our short article here.
Generally, as a rule of thumb, I advocate everyone test with as many people as they can (provided they are suitable users). Be that five, or twenty, or one hundred--what's important is to test.
The purpose of this article was to start a conversation around a topic that was taken as a given in the industry, and I hope that goal was achieved. Thanks for your comment!
I always heard 5, and never even questioned it
Same here. And then at some point, I got curious about where the 5 came from and I wanted to look into that a bit more. The result is this article.
Hope you found it useful :)