Big data goes to work

Smart companies use data to ask the right questions and take swift action.

Companies that are slow to adopt data-driven practices don’t need to worry about long-term plans — they’ll be disrupted out of existence before those deadlines arrive. And even if your business is on the data bandwagon, you shouldn’t get too comfortable. Shifts in consumer tolerances and expectations are quickly shaping how businesses apply big data.

Alistair Croll, Strata online program chair, explores these shifts and other data developments in the following interview. Many of these same topics will be discussed at “Moving to Big Data,” a free Strata Online Conference being held Dec. 7.

How are consumer expectations about data influencing enterprises?

Alistair CrollAlistair Croll: There are two dimensions. First, consumer tolerance for sharing data has gone way up. I think there’s a general realization that shared information isn’t always bad: we can use it to understand trends or fight diseases. Recent rulings by the Supreme Court and legislation like the Genetic Information Nondiscrimination Act (GINA) offer some degree of protection. This means it’s easier for companies to learn about their customers.

Second, consumers expect that if a company knows about them, it will treat them personally. We’re incensed when a vendor that claims to have a personal connection with us treats us anonymously. The pact of sharing is that we demand personalization in return. That means marketers are scrambling to turn what they know about their customers into changes in how they interact with them.

What’s the relationship between traditional business intelligence (BI) and big data? Are they adversaries?

Alistair Croll: Big data is a successor to traditional BI, and in that respect, there’s bound to be some bloodshed. But both BI and big data are trying to do the same thing: answer questions. If big data gets businesses asking better questions, it’s good for everyone.

Big data is different from BI in three main ways:

  1. It’s about more data than BI, and this is certainly a traditional definition of big data.
  2. It’s about faster data than BI, which means exploration and interactivity, and in some cases delivering results in less time than it takes to load a web page.
  3. It’s about unstructured data, which we only decide how to use after we’ve collected it and need algorithms and interactivity in order to find the patterns it contains.

When traditional BI bumps up against the edges of big, fast, or unstructured, that’s when big data takes over. So, it’s likely that in a few years we’ll ask a business question, and the tools themselves will decide if they can use traditional relational databases and data warehouses or if they should send the task to a different architecture based on its processing requirements.

What’s obvious to anyone on either side of the BI/big data fence is that the importance of asking the right questions — and the business value of doing so — has gone way, way up.

How can businesses unlock their data? What’s involved in that process?

Alistair Croll: The first step is to ask the right questions. Before, a leader was someone who could convince people to act in the absence of clear evidence. Today, it’s someone who knows what questions to ask.

Acting in the absence of clear evidence mattered because we lived in a world of risk and reward. Uncertainty meant we didn’t know which course of action to take — and that if we waited until it was obvious, all the profit would have evaporated.

But today, everyone has access to more data than they can handle. There are simply too many possible actions, so the spoils go to the organization that can choose among them. This is similar to the open-source movement: Goldcorp took its geological data on gold deposits — considered the “crown jewels” in the mining industry — and shared it with the world, creating a contest to find rich veins to mine. Today, they’re one of the most successful mining companies in the world. That comes from sharing and opening up data, not hoarding it.

Finally, the value often isn’t in the data itself; it’s in building an organization that can act on it swiftly. Military strategist John Boyd developed the observe, orient, decide and act (OODA) loop, which is a cycle of collecting information and acting that fighter pilots could use to outwit their opponents. Pilots talk of “getting inside” the enemy’s OODA loop; companies need to do the same thing.

So, businesses need to do three things:

  1. Learn how to ask the right questions instead of leading by gut feel and politics.
  2. Change how they think about data, opening it up to make the best use of it when appropriate and realizing that there’s a risk in being too private.
  3. Tune the organization to iterate more quickly than competitors by collecting, interpreting, and testing information on its markets and customers.
Moving to Big Data: Free Strata Online Conference — In this free online event, being held Dec. 7, 2011, at 9AM Pacific, we’ll look at how big data stacks and analytical approaches are gradually finding their way into organizations as well as the roadblocks that can thwart efforts to become more data driven. (This Strata Online Conference is sponsored by Microsoft.)

Register to attend this free Strata Online Conference

What are the most common data roadblocks in companies?

Alistair Croll: Everyone I talk to says privacy, governance, and compliance. But if you really dig in, it’s culture. Employees like being smart, or convincing, or compelling. They’ve learned soft skills like negotiation, instinct, and so on.

Until now, that’s been enough to win friends and influence people. But the harsh light of data threatens existing hierarchies. When you have numbers and tests, you don’t need arguments. All those gut instincts are merely hypotheses ripe for testing, and that means the biggest obstacle is actually company culture.

Are most businesses still in the data acquisition phase? Or are you seeing companies shift into data application?

Alistair Croll: These aren’t really phases. Companies have a cycle — call it a data supply chain — that consists of collection, interpretation, sharing, and measuring. They’ve been doing it for structured data for decades: sales by quarter, by region, by product. But they’re now collecting more data, without being sure how they’ll use it.

We’re also seeing them asking questions that can’t be answered by traditional means, either because there’s too much data to analyze in a timely manner, or because the tools they have can’t answer the questions they have. That’s bringing them to platforms like Hadoop.

One of the catalysts for this adoption has been web analytics, which is, for many firms, their first taste of big data. And now, marketers are asking, “If I have this kind of insight into my online channels, why can’t I get it elsewhere?” Tools once used for loyalty programs and database marketing are being repurposed for campaign management and customer insight.

How will big data shape businesses over the next few years?

Alistair Croll: I like to ask people, “Why do you know more about your friends’ vacations (through Facebook or Twitter) than about whether you’re going to make your numbers this quarter or where your trucks are?” The consumer web is writing big data checks that enterprise BI simply can’t cash.

Where I think we’ll see real disruption and adoption is in horizontal applications. The big data limelight is focused on vertical stuff today — genomics, algorithmic trading, and so on. But when it’s used to detect employee fraud or to hire and fire the right people, or to optimize a supply chain, then the benefits will be irresistible.

In the last decade, web analytics, CRM, and other applications have found their way into enterprise IT through the side door, in spite of the CIO’s allergies to outside tools. These applications are often built on “big data,” scale-out architectures.

Which companies are doing data right?

Alistair Croll: Unfortunately, the easy answer is “the new ones.” Despite having all the data, Blockbuster lost to Netflix; Barnes & Noble lost to Amazon. It may be that, just like the switch from circuits to packets or from procedural to object-oriented programming, running a data-driven business requires a fundamentally different skill set.

Big firms need to realize that they’re sitting on a massive amount of information but are unable to act on it unless they loosen up and start asking the right questions. And they need to realize that big data is a massive disintermediator, from which no industry is safe.

This interview was edited and condensed.


tags: , , , , ,