Finding the holes in qualitative and quantitative testing.
I can’t tell you how often I hear things from engineers like, “Oh, we don’t have to do user testing. We’ve got metrics.” Of course, you can almost forgive them when the designers are busy saying things like, “Why would we A/B test this new design? We know it’s better!”
In the debate over whether to use qualitative or quantitative research methods, there is plenty of wrong to go around. So, let’s look at some of the myths surrounding qualitative and quantitative research, and the most common mistakes people make when trying to use them.
Usability does not determine usefulness
I love it when companies test prototypes. Love love love it. But it makes me incredibly sad when they use prototype testing for the wrong thing.
First, let me give you my definition of “prototype testing” here. I often build interactive, or semi-interactive, prototypes when designing a product. These prototypes are not actual products. They’re simulations of products. People who see the prototype can often click around them and perform some simple tasks, but they’re generally not hooked up to a real backend system.
“Well, what good is that?” you might reasonably ask. Excellent question. Interactive prototypes are incredibly useful for finding problems in usability testing settings. In a checkout flow, you might create a simple interactive prototype and watch four or five people go through the process (with test accounts) in order to find out if there were any parts of the flow they found confusing or hard to use.
It’s a great technique for any reasonably complicated interaction that you want to test before you spend a lot of time writing code. Interactive prototype testing can save you a ton of time because it helps you make sure that you’re building the product right before you spend a lot of time and money actually writing code.
Increase your product engagement by telling users what to do.
As a consultant, I’ve talked to a lot of startups who have “social” products. You could tell that the products were “social” because they had comment sections and sharing icons that let people post to Pinterest or Facebook.
Of course, one of the things that the founders complain about is that too few users are actually making comments or sharing or doing anything remotely social with the product.
There’s a very simple reason for this: the founders have added features to their product that allow users to be social rather than encouraging them to be social.
Understanding the Difference Between User Problems, Business Needs, and Solutions
First, let me say that I love all the emphasis on Customer Development, Early User Research, and Product Market Fit that I’ve been seeing these days. What I don’t love is the massive confusion that often comes along with it.
There’s a particular type of confusion I’ve seen on teams at the very beginning of the product development process that I’d like to try to clear up. Or possibly add to. We’ll see.
Some people don’t seem to understand the difference between a Business Need, a User Problem, and a Solution. But you have to understand the difference, because if you don’t, you’ll end up doing the wrong sort of research and designing the wrong product.
A Business Need
At its very simplest, a Business Need is what a product will do for your company. This can often be expressed in the form of a metric that needs to be moved or a hypothesis about how building a new feature or product will make you a billionaire.
Here are some examples of business needs:
- Improve the conversion rate on a landing page so that we get more people trying our product.
- Increase revenue by selling more widgets.
- Get more registered users for free by getting our current users to share our product.
- Increase engagement with our product so that people are more likely to be retained users.
- Build a huge user base so that we can eventually monetize it.
What’s interesting about these Business Needs? Well, in one way or another, all of these things, if executed correctly, should eventually increase our revenue or decrease our spend. We need to do these things to have a viable business. But there are all sorts of ways to do them, some of which are great for users and others that aren’t.
To identify a business need, typically you’re going to want quantitative data. You need to know what your metrics are in order to figure out which ones need to be higher. You don’t determine a business need by talking to users.
Obviously business needs might be caused by user problems. For example, if your onboarding process is hard to use, you could have low conversion rates. But the business need is increasing the conversion rate, which you might do in a number of different ways.
A User Problem
Your users have problems. Some of the problems they’ll pay you to solve for them. Some of the problems you’re probably causing for them with your terrible UX. Some of the problems they don’t even know they have.
Here are a few examples of user problems:
- It’s hard to share documents across different computers.
- The first time experience with a particular product is confusing and complicated.
- The user can’t use an app when it’s not connected to the Internet.
- A person has trouble finding a good hair salon in her area and booking an appointment.
You’ll note that these user problems are all quite different. The first one inspired lots of companies, like DropBox. The second one is common to many products. The third one is mobile specific. The fourth one could be solved by a number of different types of products, some of which are quite low tech. There are roughly an infinite number of other user problems that could exist.
The common factor here is that these are problems experienced by humans. The other common factor is that there is no guarantee that solving a user problem will actually fulfill a business need. Sure, solving problems for people is generally a good thing, but there are some user problems that people will pay you to solve and others that they won’t.
To identify a user problem, your best bet is observational and generative research. Watch people in the wild using your product or other products. Follow people around while they perform various tasks or do their jobs. Understand the things that make life difficult for people and then identify the biggest, most important problems that you could solve for them.
A solution, as the name implies, is how you solve a problem. Ideally, your solution will solve a user problem which will fix a business need.
Here are a few examples of solutions: Read more…
Why "flow" and "context" are more important than screen size
Are we done with the Mobile First meme, yet? Can we be? Please?
Look, don’t get me wrong. I fundamentally agree with a lot of the thoughts behind the annoying catchphrase “mobile first.” For example, I agree that mobile devices are now the primary (if not only) mode of connecting for many markets. I also think that having some sort of mobile strategy is absolutely required for almost every product.
The problem is that “mobile first” often equates “mobile” with “small screen” or “responsive layout” or “native vs. mobile web.” Now, those are all incredibly important decisions. But if you’re thinking about the size of your screen or the technology you’re going to use first, you are designing wrong.
Of course, if you’ve read anything else I’ve ever written, you know that the first thing you must figure out is an important customer problem or need that your product is aimed at solving for real people. We’re going to just skip over that whole part where you get to know your most important users. But that’s always first. Promise.
Once you’ve done all that though, you need to start designing. And there are two things that you should always know before you even start considering things like screen size or technology.
Those two things are: Flow and Context.
User research you can do now
There’s a lot of advice about how to do great user research. I have some pretty strong opinions about it myself.
But, as with exercise, the best kind of research is the kind that you actually DO.
So, in the interests of getting some good feedback from your users right now, I have some suggestions for Tiny Tests. These are types of research that you could do right this second with very little preparation on your part.
What is a Tiny Test?
Tiny Tests do not take a lot of time. They don’t take a lot of money. All they take is a commitment to learning something from your users today.
Pick a Tiny Test that applies to your product and get out and run one right now. Oh, ok. You can wait until you finish the post.
Dozens of companies now exist that allow you to run an unmoderated test in a few minutes. I’ve used UserTesting.com many times and gotten some great results really quickly. I’ve also heard good things about Loop11 and several others, so feel free to pick the one that you like best.
The only thing harder to find than a great designer is a unicorn
I know, I know. Founders and entrepreneurs are already being told that they need to learn how to code, hire, raise money, and get customers.
Screw that. What founders and entrepreneurs should really do is learn how to build a useful product. And that means learning the fundamentals of research and design.
Don’t believe me? Here are six reasons you should be your own UX designer (or at least learn enough about UX design to fake it).
1. You need to know what problem you’re solving and for whom
GPS solved the problem of getting lost when going to new places. Kindle solved the problem of my entire house filling up with books I’d already read. Instagram and other similar tools solved the problem of how to share all those great photos on your phone with your friends.
Of course, not every product idea solves a problem, and not every problem is something that people are willing to pay you to solve. That’s why it’s so important to learn the fundamentals of customer development and user research.
If you know how to validate your product ideas, you’ll be able to more accurately predict which products solve important problems for large groups of people. This means that you’ll be more likely to build something that lots of people want to pay you for.