Handling UAT if users constantly miss things

Scott Hanselman asked on twitter “How do you handle user acceptance testing if the users consistently miss things?”. Sometimes it takes more than 140 characters to give a decent reply.

At my work, we run two distinct phases of testing, System Testing and User Acceptance Testing.

System Testing is the first phase, where we test that the software we’ve built meets both the requirements we were given and the specifications we wrote. This is where we exhaustively test every nook and cranny of the system to make sure it functions. Essentially we’re trying to answer the question “Did we build it right?”

User Acceptance Testing is the second phase, where real users from the business units test the software to see that it meets their needs. This is less about the minutiae of every detail and more about how the software can deliver business value. Essentially we’re trying to answer the question “Did we build the right thing?”

Taking this approach means that it’s less critical if our business users miss things during testing, because we’re not expecting them to exhaustively test every line of code. We’ve already done that ourselves.

Our key realisation is that our business users aren’t experts in testing, they’re experts in their own business. Asking them to do a professional job of something they’re not trained to do seems to be somewhat unfair.

One more thing - we don’t wait for UAT to get our first feedback that we’re building the right thing. We have subject matter experts and business analysis heavily involved the entire time. UAT includes the signoff milestone that, having received what they wanted built, the business unit is now able to achieve the outcomes desired.