I’m finishing up a 3 month beta test of Stacks 3. This post is a reminder to future-me when I begin my next one. It’s tough to get my developer-brain-assumptions into the right frame of mind for collecting solid info from novice users.
During a public beta I make it a regular habit to explain that the goal of beta testing is to root out problems so that I can fix them for everyone. For many novice users this is a new concept. Their experiences with tech support up to this point is usually quite different.
Normal Tech Support
Tech support is contacted when a user has trouble. The goal of tech support is to eliminate or work around the trouble for that one user. Usually this is done by showing the user how to use the product or work around a known problem.
In other words: I tell the user they’re holding it wrong and teach them how to hold it right.
Bug reporting and Beta Testing
A bug report starts exactly the same: the user has trouble. But that’s where the similarities stop. The goal of a bug report is to fix the bug, not just for one user, but for everyone. This is achieved by isolating and examining the bug, then fixing it. Any changes I ask the user to make will make the bug worse, not better, to make examining it easier.
In other words: I ask the user to show me how they’re holding it wrong so that I can hold it wrong too.
Dear Future Me
In a public beta most users will contact you expecting help. Communicating your bug-fixing goals early and in a friendly way is key. But communicating your goals won’t change theirs – they’ll still want to work around the problem and get back to productivity. When you’ve got your bug report, remember to return the favor and help your beta testers fix things and keep working.