Building the right thing vs. building the thing right

Usability does not determine usefulness

I love it when companies test prototypes. Love love love it. But it makes me incredibly sad when they use prototype testing for the wrong thing.

First, let me give you my definition of “prototype testing” here. I often build interactive, or semi-interactive, prototypes when designing a product. These prototypes are not actual products. They’re simulations of products. People who see the prototype can often click around them and perform some simple tasks, but they’re generally not hooked up to a real back end system.

“Well, what good is that?” you might reasonably ask. Excellent question. Interactive prototypes are incredibly useful for finding problems in usability testing settings. In a checkout flow, you might create a simple interactive prototype and watch four or five people go through the process (with test accounts) in order to find out if there were any parts of the flow they found confusing or hard to use.

It’s a great technique for any reasonably complicated interaction that you want to test before you spend a lot of time writing code. Interactive prototype testing can save you a ton of time because it helps you make sure that you’re building the product right before you spend a lot of time and money actually writing code.

However, the thing I see all the time is people using prototypes to try to validate that they’re building the right product. Stop it. Stop it right now.

Let me explain. The common pattern is that people go out and do some research on a problem (yay!) and then their next step is to build a prototype of their solution in order to show it to people and get feedback.

This is a wonderful thing to do. You should absolutely show your prototype to people in order to get feedback. You just shouldn’t expect feedback about whether or not anybody will actually buy or use your product. In other words, you can’t use a prototype to learn if you are building the right product.

Why not? Well, it has to do with the nature of doing qualitative research on a prototype.

Imagine that you ask somebody to look at your product. Maybe you stopped them on the street. Maybe they answered an ad on Craigslist. Maybe it’s a friend or a friend of a friend. You ask them a few questions about their life or job. Then you show them something that looks kind of like a product. They can click around and input some data. They can perform a few tasks. Then you ask them if they would use this product when it becomes available in a couple of months or weeks.

They will overwhelmingly say, “yes.” If they don’t say yes, they will say, “probably,” or, “no, but my cousin would use it.” You will get a huge number of false positives for all of the following reasons:

  • people want to be polite and helpful
  • people dramatically overestimate their willingness to adopt new things
  • people are bad at predicting the future
  • people are bad at extrapolating real product behavior from an interactive prototype

None of these problems come into play when you’re using a prototype to just test for usability, because usability testing is about evaluating facts. It answers questions like:

  • Will a reasonable person be able to perform a specific task?
  • Will a normal person get confused by the messaging?
  • Am I building this product right?

Testing for usefulness, on the other hand, is about trying answer questions like:

  • Will a lot of people buy my product at some uncertain point in the future?
  • Is this product the best solution to a problem?
  • Am I building the right product?

And these are simply not things you can answer with prototype testing.

But don’t worry. All hope is not lost. You can still test whether you’ve got the right solution to problems before you build an entire product. You just need to validate in other ways. Important ways. Ways I’m not going to go into right now because this is already unconscionably long. I covered a few of the key points in my talk at the Lean Startup Conference in 2013 though, and you can see a video of that here.

tags:

Get the O’Reilly Programming Newsletter

Weekly insight from industry insiders. Plus exclusive content and offers.