Finding the holes in qualitative and quantitative testing.
I can’t tell you how often I hear things from engineers like, “Oh, we don’t have to do user testing. We’ve got metrics.” Of course, you can almost forgive them when the designers are busy saying things like, “Why would we A/B test this new design? We know it’s better!”
In the debate over whether to use qualitative or quantitative research methods, there is plenty of wrong to go around. So, let’s look at some of the myths surrounding qualitative and quantitative research, and the most common mistakes people make when trying to use them.
Winning organizations continually experiment.
I constantly hear how enterprises are poor at innovation, bad at product development and unresponsive to business change. So it begs the question, why do so many organizations get it wrong? And what are the key factors to consider when trying to innovate in large organizations?
Typically the factors constraining innovation are conflicting business goals, competing priorities, localized performance measures and success criteria. While these have traditionally been the tools of management — to control workforce behavior and output — in highly competitive and quickly evolving business environments they also have had the adverse effects of killing creativity, responsiveness and ingenuity.
So what are the components needed to unleash innovation in enterprise?
Learn to resist vanity metrics
One of the things we preach in Lean Analytics is that entrepreneurs should avoid vanity metrics—numbers that make you feel good, but ultimately, don’t change your behavior. Vanity metrics (such as “total visitors”) tend to go “up and to the right” but don’t tell you much about how you’re doing.
Many people find solace in graphs that go up and to the right. The metric “Total number of people who have visited my restaurant” will always increase; but on its own it doesn’t tell you anything about the health of the business. It’s just head-in-the-sand comforting.
A good metric is often a comparative rate or ratio. Consider what happens when you put the word “per” before or after a metric. “Restaurant visitors per day” is vastly more meaningful. Time is the universal denominator, since the universe moves inexorably forwards. But there are plenty of other good ratios. For example, “revenue per restaurant visitor” matters a lot, since it tells you what each diner contributes.
What’s an active user, anyway?
For many businesses, the go-to metric revolves around “active users.” In a mobile app or software-as-a-service business, only some percentage of people are actively engaged. In a media site, only some percentage uses the site each day. And in a loyalty-focused e-commerce company, only some buyers are active.
This is true of more traditional businesses, too. Only a percentage of citizens are actively engaged in local government; only a certain number of employees are using the Intranet; only a percentage of coffee shop patrons return daily.
Unfortunately, saying “measure active users” begs the question: What’s active, anyway?
To figure this out, you need to look at your business model. Not your business plan, which is a hypothetical projection of how you’ll fare, but your business model. If you’re running a lemonade stand, your business model likely has a few key assumptions:
- The cost of lemonade;
- The amount of foot traffic past your stand;
- The percent of passers-by who will buy from you;
- The price they are willing to pay.
Our Lean lemonade stand would then set about testing and improving each metric, running experiments to find the best street corner, or determine the optimal price.
Lemonade stands are wonderfully simple, so your business may have many other assumptions, but it is essential that you quantify them and state them so you can then focus on improving them, one by one, until your business model and reality align. In a restaurant, for example, these assumptions might be, “we will have at least 50 diners a day” or “diners will spend on average $20 a meal.”
The activity you want changes
We believe most new companies and products go through five distinct stages of growth:
- Empathy, where you figure out what problem you’re solving and what solution people want;
- Stickiness, where you measure how many people adopt your solution rather than trying it and leaving;
- Virality, where you maximize word-of-mouth and references;
- Revenue, where you pour some part of your revenues back into paid acquisition or advertising;
- Scale, where you grow the business through automation, delegation, and process.
Understanding the Difference Between User Problems, Business Needs, and Solutions
First, let me say that I love all the emphasis on Customer Development, Early User Research, and Product Market Fit that I’ve been seeing these days. What I don’t love is the massive confusion that often comes along with it.
There’s a particular type of confusion I’ve seen on teams at the very beginning of the product development process that I’d like to try to clear up. Or possibly add to. We’ll see.
Some people don’t seem to understand the difference between a Business Need, a User Problem, and a Solution. But you have to understand the difference, because if you don’t, you’ll end up doing the wrong sort of research and designing the wrong product.
A Business Need
At its very simplest, a Business Need is what a product will do for your company. This can often be expressed in the form of a metric that needs to be moved or a hypothesis about how building a new feature or product will make you a billionaire.
Here are some examples of business needs:
- Improve the conversion rate on a landing page so that we get more people trying our product.
- Increase revenue by selling more widgets.
- Get more registered users for free by getting our current users to share our product.
- Increase engagement with our product so that people are more likely to be retained users.
- Build a huge user base so that we can eventually monetize it.
What’s interesting about these Business Needs? Well, in one way or another, all of these things, if executed correctly, should eventually increase our revenue or decrease our spend. We need to do these things to have a viable business. But there are all sorts of ways to do them, some of which are great for users and others that aren’t.
To identify a business need, typically you’re going to want quantitative data. You need to know what your metrics are in order to figure out which ones need to be higher. You don’t determine a business need by talking to users.
Obviously business needs might be caused by user problems. For example, if your onboarding process is hard to use, you could have low conversion rates. But the business need is increasing the conversion rate, which you might do in a number of different ways.
A User Problem
Your users have problems. Some of the problems they’ll pay you to solve for them. Some of the problems you’re probably causing for them with your terrible UX. Some of the problems they don’t even know they have.
Here are a few examples of user problems:
- It’s hard to share documents across different computers.
- The first time experience with a particular product is confusing and complicated.
- The user can’t use an app when it’s not connected to the Internet.
- A person has trouble finding a good hair salon in her area and booking an appointment.
You’ll note that these user problems are all quite different. The first one inspired lots of companies, like DropBox. The second one is common to many products. The third one is mobile specific. The fourth one could be solved by a number of different types of products, some of which are quite low tech. There are roughly an infinite number of other user problems that could exist.
The common factor here is that these are problems experienced by humans. The other common factor is that there is no guarantee that solving a user problem will actually fulfill a business need. Sure, solving problems for people is generally a good thing, but there are some user problems that people will pay you to solve and others that they won’t.
To identify a user problem, your best bet is observational and generative research. Watch people in the wild using your product or other products. Follow people around while they perform various tasks or do their jobs. Understand the things that make life difficult for people and then identify the biggest, most important problems that you could solve for them.
A solution, as the name implies, is how you solve a problem. Ideally, your solution will solve a user problem which will fix a business need.
Here are a few examples of solutions: Read more…
How fast can the enterprise change?
At last year’s Fluent Conference, I kept having the same conversation with attendees from large companies. They had come to the show with a mandate from their bosses to figure out how to bring that fast-moving web work into their slow-moving enterprise systems.
I enjoyed some of that same conversation this year, but also a different note: even with management support, making that transition was difficult. Some parts meshed, others were difficult: changes in one place could reverberate through many others. The whole concept of “rapid prototyping” fit badly with a variety of technologies and approaches meant to minimize unpleasant surprises. Even eight years after the advent of Ajax, a variety of server-centric techniques limit the flexibility of front-end developers.
Someone at lunch said that “the technology helps, but the culture matters.” A few others talked about how everyone wanted better front-end work, but thought it could be grafted easily on existing back-end practice. The shiny parts are easy to talk about, but the plumbing is harder.
I was happy to see Bill Scott (@billwscott) of PayPal take on these challenges in his keynote. Bill wasn’t smuggling anything—that would be difficult under the title “Clash of the Titans: Releasing the Kraken.” He was brought to PayPal to change the company, to bring the lean “build – measure – learn” approach. In a risk-averse world, with “a 20-day class on how to use their version of Spring,” Scott had to change the “culture of a long shelf life” (something publishing folks are starting to do as well).
It’s a hard-hitting talk, calling for major change, skunkworks projects, and shifts in both company culture and technology.
User research you can do now
There’s a lot of advice about how to do great user research. I have some pretty strong opinions about it myself.
But, as with exercise, the best kind of research is the kind that you actually DO.
So, in the interests of getting some good feedback from your users right now, I have some suggestions for Tiny Tests. These are types of research that you could do right this second with very little preparation on your part.
What is a Tiny Test?
Tiny Tests do not take a lot of time. They don’t take a lot of money. All they take is a commitment to learning something from your users today.
Pick a Tiny Test that applies to your product and get out and run one right now. Oh, ok. You can wait until you finish the post.
Dozens of companies now exist that allow you to run an unmoderated test in a few minutes. I’ve used UserTesting.com many times and gotten some great results really quickly. I’ve also heard good things about Loop11 and several others, so feel free to pick the one that you like best.
The only thing harder to find than a great designer is a unicorn
I know, I know. Founders and entrepreneurs are already being told that they need to learn how to code, hire, raise money, and get customers.
Screw that. What founders and entrepreneurs should really do is learn how to build a useful product. And that means learning the fundamentals of research and design.
Don’t believe me? Here are six reasons you should be your own UX designer (or at least learn enough about UX design to fake it).
1. You need to know what problem you’re solving and for whom
GPS solved the problem of getting lost when going to new places. Kindle solved the problem of my entire house filling up with books I’d already read. Instagram and other similar tools solved the problem of how to share all those great photos on your phone with your friends.
Of course, not every product idea solves a problem, and not every problem is something that people are willing to pay you to solve. That’s why it’s so important to learn the fundamentals of customer development and user research.
If you know how to validate your product ideas, you’ll be able to more accurately predict which products solve important problems for large groups of people. This means that you’ll be more likely to build something that lots of people want to pay you for.