Editor’s note: this is an excerpt from our forthcoming book Designing with Data; it is part of a free curated collection of chapters from the O’Reilly Design library — download a free copy of the Experience Design ebook here.
The phrase “data driven” has long been part of buzzword-bingo card sets. It’s been heard in the halls of the web analytics conference eMetrics for more than a decade, with countless sessions aimed at teaching audience members how to turn their organizations into data-driven businesses.
When spoken of in a positive light, the phrase data driven conjures visions of organizations with endless streams of silver-bullet reports — you know the ones: they’re generally entitled something to the effect of “This Chart Will Help Us Fix Everything” and show how a surprise change can lead to a quadrillion increase in revenue along with world peace.
When spoken of in a negative light, the term is thrown around as a descriptor of Orwellian organizations with panopticon-level data collection methods, with management imprisoned by relentless reporting, leaving no room for real innovation.
Evan Williams, founder of Blogger, Twitter, and Medium, made an apt comment about being data driven:
I see this mentality that I think is common, especially in Silicon Valley with engineer-driven start-ups who think they can test their way to success. They don’t acknowledge the dip. And with really hard problems, you don’t see market success right away. You have to be willing to go through the dark forest and believe that there’s something down there worth fighting the dragons for, because if you don’t, you’ll never do anything good. I think it’s kind of problematic how data-driven some companies are today, as crazy as that sounds.”
One thing is certain: businesses can’t do LinkedIn-level data-driven decision-making on a shoe-string budget. As we’ll see later on in this chapter, real data-driven decision-making isn’t even possible at certain stages in the development of a company. Early-stage companies have, in most cases, far more pressing issues than split testing the color of a button.
One of the best descriptions that we’ve ever seen on the difference between data-driven and data-informed comes by way of Andrew Chen. In a post entitled “Know the difference between data-informed and data-driven,” he explains that “the difference…in my mind, is that you weigh the data as one piece of a messy problem you’re solving with thousands of constantly changing variables. While data is concrete, it is often systematically biased. It’s also not [always] the right tool, because not everything is an optimization problem. And delegating your decision-making to only what you can measure right now often de-prioritizes more important macro aspects of the problem.”
The words “not everything is an optimization problem” sums up the philosophy behind this book very well. We also want to broaden the definition of “data” from merely being AB test results to representing a wide breadth of information you can gather from many different sources. We want to acknowledge that the intersection of data and design can be so much more than AB testing and to recognize that it can be equal parts art and science.
To be honest, the great debate between “data-driven” vs. “data-informed” can sometimes feel like semantics. It is never as black and white as the blogs might have us believe. There are people who call themselves “data-driven,” but in practice, exercise judgment and gut to inform their decisions as much as the data; even if you place yourself in the data-informed camp, if you aren’t using the data correctly, then you may run as much risk of doing as much or even more damage than someone who religiously uses data to drive every single decision. We generally felt, though, that data-informed better represented the balance we are trying to strike, and this is why data-driven design wasn’t the title we chose for this book.
What is data-informed design?
Data-informed design is an approach to design where decisions are informed by specific, objective evidence: data. And though data comes in many forms, and it is collected in a number of different ways, it should be used to give teams a better understanding of what customers are doing with a product and how they are reacting to the changes made to it.
Being smart about data-informed decision-making has considerable advantages. It helps managers hire smarter, and having common success metrics within your company can also help designers and the broader product team to align around common goals and to understand what kind of data is the most important to track and follow.
So, more specifically for our purposes, data-informed design is a framework that you can use as a designer to help you hone your understanding of customer behavior, and align you and your team to the larger company objectives and business goals.
Using a data-informed approach can be a core foundational practice and primary methodology for how you and your teams can think about design — but as with most things in life, it’s not all or nothing. There are other things to take into consideration when designing as well.
There is a general perception that taking a data-informed approach will limit your ability to take large leaps in innovation or that it stifles creativity. Similarly, many raise the concern that using a data-driven approach limits designers by taking the art out of design, reducing it to just the science. We believe that it is most important to recognize that solely employing a data-driven approach will not be enough. For us, being data-informed is less about being a “yes” or “no” question, and it’s much more about being aware that there is a spectrum on which you can operate. Depending on the nature of the project and where you are in the development timeline and the product itself, it may make more or less sense to rely on data. Generally speaking, when people talk about the opposite of data informed, they mean relying on experience, gut, or instinct — or making decisions in absence of having them be vetted by data.
It’s important to recognize applying a data-informed framework is not one size fits all.We believe that relying on your instinct and experience is essential during the early stages of a product. When you find that you are trying to pivot or make a dramatic change in your product direction, those initial ideas and product solutions are born from your instinct and experience as a designer. When you are defining principles for your brand or product and thinking about how you want to express these beliefs to your customers, you again rely more on your core beliefs, instincts, and experience as a designer.
A significant part of a product designer’s job is still the art and craft of design. That is leveraging training, experience, instinct, and gut based on years of perfecting the craft of creating great experiences in terms of both visual treatment and user experience (flows, information architecture). If you take two designers, one who is mediocre and one who is great, and they both employ a data-driven framework, you’ll find that the better designer will still be able to propose stronger and more impactful solutions compared to the mediocre designer. However, in a data-informed framework, you might actually be able to measure how significant that difference is.
Now that we’ve outlined our basic philosophy on data-informed design, it’s important to recognize that applying this framework is not one size fits all. A key fundamental consideration is finding a baseline before you even start to optimize and tune your approach. The type of business you’re in and the stage that your business is in will matter tremendously.
This post is part of our ongoing look at experience design’s role in the business world.