Big Data Culture Gap: Technology Advancing More Quickly Than People and Processes

Way back in October 2012, mere days before Hurricane Sandy filled our basement with five feet of water, the nice editors at O’Reilly Media asked me to write a white paper on the emerging architecture of big data. That paper was followed by two more, one about the emerging culture of big data, and another about the impact of big data on the traditional IT function.

Despite my attempts to make all three papers seem wildly different, they all share a common theme or subtext, namely that the technology of big data is evolving far more quickly than the people and processes of big data.

In other words, the tools are more advanced than the organizations using them. At least that’s my takeaway. After interviewing dozens of data analysts, industry experts, and senior-level corporate executives, I’m convinced that the technology is advancing faster than the abilities of the people trying to use it.

In retrospect, it’s not surprising that the technology of big data has evolved more rapidly than the organizational structures required to harness big data and convert it into a steady source of value. C-level executives and boards of directors regard big data as promising, but they weren’t born yesterday, and they need to see the business case before they start writing big checks. “Tactics in search of a strategy” is how one senior executive recently summed up his thoughts on big data.

Many of us sense that big data is destined to become the next big thing, but none of us is quite sure how it will all play out.

We can take comfort in the knowledge that this has all happened before. When airplanes were first pressed into military service, they were used exclusively for reconnaissance. When a team of engineers led by Anthony Fokker developed the synchronized machine gun, airplanes morphed into lethal weapons and a new strategy—aerial warfare—was born.

Most of us know the story of how the folks at Xerox didn’t understand the value of the clunky computer mouse they had invented, and how Steve Jobs took the basic idea and engineered it into a practical tool that helped launch a revolution in personal computing.

Based on what I’ve been hearing from my sources, I have the feeling that we are five or six inventions away from a similar revolution in big data. I’m not really sure what form those inventions will take—or I would quit my day job and invest in the companies making them—but I am certain that multiple disciplines and technologies will be involved.

When it arrives, the big data revolution will transform the modern enterprise, accelerate the growth of markets, and launch a new era of social commerce. The changes—particularly in emerging economies—will be mind-boggling in both scale and scope.

Ten years from now, we’re going to look back and wonder why we didn’t see it coming, but by then we’ll be on to the next big thing, and big data will seem about as interesting as a laptop from the 1990s. Meantime, encourage your kids to learn Python, Ruby, and R.

Cheers!

Download The Culture of Big Data for free.