This has been going on for a long time, likely since the invention of fire, knives, or the printed word. But I want to focus specifically on computing technology. The human race is busy colonizing a second online world and sticking prosthetic brains — today, we call them smartphones — in front of our eyes and ears. And stacks of technology on which they rely are vulnerable.

When we first created automatic phone switches, hackers quickly learned how to blow a Cap’n Crunch whistle to get free calls from pay phones. When consumers got modems, attackers soon figured out how to rapidly redial to get more than their fair share of time on a BBS, or to program scripts that could brute-force their way into others’ accounts. Eventually, we got better passwords and we fixed the pay phones and switches.

We moved up the networking stack, above the physical and link layers. We tasted TCP/IP, and found it good. Millions of us installed Trumpet Winsock on consumer machines. We were idealists rushing onto the wild open web and proclaiming it a new utopia. Then, because of the way the TCP handshake worked, hackers figured out how to DDOS people with things like SYN attacks. Escalation, and router hardening, ensued.

We built HTTP, and SQL, and more. At first, they were open, innocent, and helped us make huge advances in programming. Then attackers found ways to exploit their weaknesses with cross-site scripting and buffer overruns. They hacked armies of machines to do their bidding, flooding target networks and taking sites offline. Technologies like MP3s gave us an explosion in music, new business models, and abundant crowd-sourced audiobooks — even as they leveled a music industry with fresh forms of piracy for which we hadn’t even invented laws.

Here’s a more specific example of unintended consequences. Paul Mockapetris is one of the creators of today’s Internet. He created DNS and implemented SMTP, fundamental technologies on which all of us rely. But he’s also single-handedly responsible for all the spam in the world.

That might be a bit of an overstatement, though I tease him about it from time to time. But there’s a nugget of truth to it: DNS was a simplified version of more robust directories like those in X.25. Paul didn’t need all that overhead, because he was just trying to solve the problem of remembering all those Internet addresses by hand, not trying to create a hardened, authenticated, resilient address protocol. He also created SMTP, the Simple Mail Transport Protocol. It was a whittled-down version of MTP — hence the “S” — and it didn’t include end-to-end authentication.

These two things — SMTP and DNS — make spam possible. If either of them had some kind of end-to-end authentication, it would be far harder for spammers to send unwanted messages and get away with it. Today, they’re so entrenched, that attempts to revise email protocols in order to add authentication have consistently failed. We’re willing to live with the glut of spam that clogs our servers because of the tremendous value of email.

We owe much of the Internet’s growth to simplicity and openness. Because of how Paul built DNS and SMTP, there’s no need to go through a complex bureaucracy to start something, or to jump through technical hoops to send an email to someone you met at a bar. We can invite a friend to a new application without strictures and approvals. The Internet has flourished precisely because it was built on a foundation of loose consensus and working code. It’s also flourished in spite of it.

Each of these protocols, from the lowly physical connections and links of Ethernet and PPP all the way up through TCP sessions and HTTP transactions, are arranged in a stack, independent layers of a delicious networking cake. By dividing the way the Internet works into discrete layers, innovation can happen at one layer (copper to fiber; Token Ring to Ethernet; UDP to TCP; Flash to DHTML; and so on) independent of the other layers. We didn’t need to rewrite the Internet to build YouTube.

Paul, and the other framers of the web, didn’t know we’d use it to navigate, or stream music — but they left it open so we could. But where the implications of BBS hacking or phone phreaking were limited to a small number of homebrew hackers, the implications for the web were far bigger, because by now, everyone relied on it.

Anyway, on to big data.

Geeks often talk about “layer 8.” When an IT operator sighs resignedly that it’s a layer 8 problem, she means it’s a human’s fault. It’s where humanity’s rubber meets technology’s road. And big data is interesting precisely because it’s the layer 8 protocol. It’s got great power, demands great responsibility, and portends great risk unless we do it right. And just like the layers beneath it, it’s going to get good, then bad, then stable.

Other layers of the protocol stack have come under assault by spammers, hackers, and activists. There’s no reason to think layer 8 won’t as well. And just as hackers find a clever exploit to intercept and spike an SSL session, or trick an app server into running arbitrary code, so they’ll find an exploit for big data.

The term “data warfare” might seem a bit hyperbolic, so I’ll try to provide a few concrete examples. I’m hoping for plenty more in the Strata Online Conference we’re running next week, which has a stellar lineup of people who have spent time thinking about how to do naughty things with information at scale.

Injecting noise

Analytics applications rely on tags embedded in URLs to understand the nature of traffic their receive. URLs contain parameters that identify the campaign, the medium, and other information on the source of visitors. For example, the URL http://www.solveforinteresting.com?utm_campaign=datawar tells Google Analytics to assign visits from that link to the campaign “datawar.” There’s seldom any verification of this information — with many analytics packages it’s included in plain text. Let’s say, as a joke, you decide you’d like your name to be the most prolific traffic source on a friend’s blog. All you need is a few willing participants, and you can simply visit the blog from many browsers and machines using your name as the campaign tag. You’ll be the top campaign traffic source.

This seems innocent enough, until you realize that you can take a similar approach to misleading your competitor. You can make them think a less-effective campaign is outperforming a successful one. You can trick them into thinking Twitter is a better medium than Google+, when in fact the reverse is true, which causes them to pay for customer acquisition in less-effective channels.

The reality isn’t this simple — smart businesses track campaigns by outcomes such as purchases rather than by raw visitors. But the point is clear: open-ended data schemes like tagging work because they’re extensible and simple, but that also makes them vulnerable. The practice of “googlebombing,” is a good example. Linking a word or definition to a particular target (such as sending searches for “miserable failure” to a biography on the White House website) simply exploits the openness of Google’s underlying algorithms.

But even if you think you have a reliable data source, you may be wrong. Consider that a few years ago, only 324 Athenians reported having swimming pools on their tax returns. This seemed low to some civil servants in Greece, so they decided to check. Using Google Map, they counted 16,974 of them  — despite efforts by citizens to camouflage their pools under green tarpaulins.

Whether the data is injected, or simply collected unreliably, data’s first weakness is its source. Collection is seldom authenticated. There’s a reason prosecutors insist on chain of evidence; but big data and analytics, like DNS and SMTP, is usually built for simplicity and ubiquity rather than for resiliency or auditability.

Mistraining the algorithms

Most of us get attacks almost daily, in the form of spam and phishing. Most of these attacks are blocked by heuristics and algorithms.

Spammers are in a constant arms race with these algorithms. Each message that’s flagged as spam is an input into anti-spam algorithms — so if a word like “Viagra” appears in a message you consider spam, then the algorithm is slightly more likely to consider that word “spammy” in future.

If you run a blog, you probably see plenty of comment spam filled with nonsense words — these are attempts to mistrain the machine-learning algorithms that block spammy content by teaching it innocuous words, undermining its effectiveness. You’re actually watching a fight between spammers and blockers, played out comment by comment, on millions of websites around the world.

Making other attacks more effective

Anti-spam heuristics happen behind the scenes, and they work pretty well. Despite this, some spam does get through. But when it does, we seldom click on it, because it’s easy to spot. It’s poorly worded; it comes from an unfamiliar source; it doesn’t render properly in our mail client.

What if that weren’t the case?

A motivated attacker can target an individual. If they’re willing to invest time researching their target, they can gain trust or impersonate a friend. The discovery of several nation-state-level viruses aimed at governments and rich targets shows a concerted, hand-crafted phishing attack can work. In the hands of an attacker, tools like Facebook’s Graph Search or Peekyou are a treasure trove of facts that can be used to craft a targeted attack.

The reason spam is still easy to spot is that it’s traditionally been hard to automate this work. People don’t dig through your trash unless you’re under investigation.

Today, however, consumers have access to “big data” tools that spy agencies could only dream of a few short years ago, which means attackers do, too, and the effectiveness of phishing, identity theft, and other information crimes will soar once bad actors learn how to harness these tools.

But digging through virtual trash and data exhaust is what machines do best. Big data lets personal attacks work at scale. If smart data scientists with decent grammar tried to maximize spam effectiveness, we’d lose quickly. To them, phishing is just another optimization problem.

Trolling to polarize

Data warfare doesn’t have to be as obvious as injecting falsehoods, mis-training machine learning algorithms, or leveraging abundant personal data to trick people. It can be far more subtle. Let’s say, for example, you wanted to polarize a political discussion such as gun control in order to reduce the reasoned discourse of compromise and justify your hard-lined stance. All you need to do is get angry.

A recent study showed that the tone of comments in a blog post had a tangible impact on how readers responded to the post. When comments used reasonable language, readers’ views were more moderate. But when those comments were aggressive, readers hardened their positions. Those that agreed with the post did so more strongly, and those who disagreed objected more fiercely. The chance for compromise vanished.

Similar approaches can sway sentiment analysis tools that try to gauge public opinion on social platforms. Once reported, these sentiments often form actual opinion, because humans like to side with the majority. Data becomes reality. There are plenty of other examples of “adversarial” data escalation. Consider the programmer who created 800,000 books and injected them into Amazon’s catalog.  Thanks to the frictionless nature of ebooks and the ease of generating them, he’s saturated their catalog (hat tip to Edd for this one.)

The year of data warfare

Data warfare is real. In some cases, such as spam, it’s been around for decades. In other cases, like tampering with a competitor’s data, it’s been possible, but too expensive, until cloud computing and new algorithms made it cheap and easy. And in many new instances, it’s possible precisely because of our growing dependence on information to lead our daily lives.

Just as the inexorable cycle of good, bad, and stable has happened at every layer, so it will happen with big data. But unlike attacks on lower levels of the stack, this time it won’t just be spam in an inbox. It’ll be both our online and offline lives. Attackers can corrupt information, blind an algorithm, inject falsehood, changing outcomes in subtle, insidious ways that undermine a competitor or flip an election. Attacks on data become attacks on people.

If I have to pick a few hot topics for 2013, data warfare is one of them. I’m looking forward to next week’s online event, because I’m convinced that this arms race will affect all of us in the coming years, and it’ll be a long time before the armistice of détente.

This post originally appeared on Strata.

“Do you know why the French hate traffic cameras?” he asked me. “It’s because it makes it hard for them to cheat on their spouses.”

He contended that while it was possible for a couple to overlook subtle signs of infidelity — a brush of lipstick on a collar, a stray hair, or the smell of a man’s cologne — the hard proof of a speeding ticket given on the way to an afternoon tryst couldn’t be ignored.

Humans live in these grey areas. A 65 mph speed limit is really a suggestion; it’s up to the officers to enforce that limit. That allows for context: a reckless teen might get pulled over for going 70, but a careful driver can go 75 without incident.

But a computer that’s programmed to issue tickets to speeders doesn’t have that ambiguity. And its accusations are hard to ignore because they’re factual, rooted in hard data and numbers.

Did big data kill privacy?

With the rise of a data-driven society, it’s tempting to pronounce privacy dead. Each time we connect to a new service or network, we’re agreeing to leave a digital breadcrumb trail behind us. And increasingly, not connecting makes us social pariahs, leaving others to wonder what we have to hide.

But maybe privacy is a fiction. For millennia — before the rise of city-states — we lived in villages. Gossip, hearsay, and whisperings heard through thin-walled huts were the norm.

Shared moral values and social pressure helped groups to compete better against other groups, helping to evolve the societies and religions that dominate the world today. Humans thrive in part because of our groupish nature — which is why moral psychologist Jonathan Haidt says we’re 90% chimp and 10% bee. We might have evolved as selfish individuals, but we conquered the Earth as selfish teams.

In other words, being private is relatively new, perhaps only transient, and gossip helped us get here.

Prediction isn’t invasion

Much of what we see as technology’s invasion of privacy is really just prediction. As we connect the world’s databases — tying together smartphones, loyalty programs, medical records, and the other constellations in the galaxy of our online lives — we’re doing something that looks a lot like invading privacy. But it’s not.

Big data doesn’t peer into your browser history or look through your bedside table to figure out what porn you like; rather, it infers your taste in smut from the kind of music you like. Big data doesn’t administer a pregnancy test; instead, it guesses you’re pregnant because of what you buy. Many of big data’s predictions are a boon, helping us to fight disease, devote resources to the right problems, and pinpoint ways to help the disadvantaged.

Is prediction an invasion of privacy? Not really. Companies will compete based on their ability to guess what’s going to happen. We’re simply taking the inefficiency out of the way we’ve dealt with risk in the past. Algorithms can be wrong. Prediction is only a problem when we cross the moral Rubicon of prejudice: treating you differently because of those predictions, changing the starting conditions for unfair reasons.

Unfortunately, big data’s predictions are often frighteningly accurate, so the temptation to treat them as fact is almost overwhelming. Policing looks like thoughtcrime. And tomorrow, a just society is a skeptical one.

We’re leakier than we know

Long before the Internet, we left a breadcrumb trail of personal details behind us: call history, credit-card receipts, car mileage, bank records, music purchases, library check-outs, and so on.

But until big data, baking the breadcrumbs back into a loaf was hard. Paper records were messy, and physical copies were hard to collect. Unless you were being pursued by an army of investigators, the patterns of your life remained hidden in plain sight. We weren’t really private — we just felt like we were, and it was too hard for others to prove otherwise without a lot of work.

No more. Big data represents a radical drop in the cost of tying together vast amounts of disparate data quickly. Digital records are clean, easy to analyze, and trivial to copy. That means the illusion of personal privacy is vanishing — but we should remember that it’s always been an illusion.

Our digital lives make this even more true. We’re probably not aware of what’s being collected as we surf the web — but it’s pretty easy to tell where someone’s been through browser trickery, cross-site advertising, and the like. So when a politician calls for your vote, they may know more about you than you want. But let’s not confound promiscuous surfing behavior — leaving more breadcrumbs — with an improved ability to bake those crumbs back into a loaf.

Big data didn’t force us to overshare; it’s just better at noticing when we do and deriving meaning from it. And because of this, it’s back to thin-walled huts and gossip. Only this time, because it’s digital and machine-driven, there are a couple of important twists to consider.

This ain’t your ancestors’ privacy

There are two key differences, however, between our ancestors’ gossip-filled, thin-walled villages and today’s global digital village.

First, consider the two-way flow of gossip. A thousand years ago, word-of-mouth worked both ways. Someone who told tales too often risked ostracism. We could confront our accusers. Social mores were a careful balance of shame and approval, with checks and balances.

That balance is gone. We can’t confront our digital accusers. If we’re denied a loan, we lack the tools to understand why. Often, we aren’t even aware that we’ve been painted with a digital scarlet letter. As one Oxford professor put it, “nobody knows the offer they didn’t receive.”

Big data is whispering things about us — both inferred predictions and assembled truths — and we don’t even know it.

Second, everyone knew gossip was imperfect. We’ve all played “broken telephone” and seen how easily many mouths distort a message. We’re skeptical of a single truth. We’ve learned to forgive, to question.

The same studies that show groups should ostracize those who don’t chip in also suggest that the best strategy of all is to forgive occasionally — just in case the initial failure was an honest mistake. In other words, when dealing with whispered truths, we lived life in a grey area.

Unfortunately, digital accusations — like those made by traffic cameras — leave little room for mercy and tolerance because they lack that grey area in which much of human interaction thrives. If we’re going to build data-driven systems, then those systems need grey areas.

New rules for the new transparency

In the timeline of human history, privacy is relatively recent. It may even be that privacy was an anomaly, that our social natures rely on leakage to thrive, and that we’re nearing the end of a transient time where the walls between us gave us the illusion of secrecy.

But now that technology is tearing down those walls, we need checks and balances to ensure that we don’t let predictions become prejudices. Even when those predictions are based in fact, we must build both context and mercy into the data-driven decisions that govern our quantified future.

This post originally appeared on Solve for Interesting. This version has been lightly edited.

Photos: Traffic camera, Silver Spring – 16th St Circle Traffic Camera by thisisbossi, on Flickr; leaky cup, The cup that can only be half-full. by vrogy, on Flickr.

Related:

• New ethics for a new world
• Big data is our generation’s civil rights issue, and we don’t know it

On its surface, a data-driven society is more transparent, and makes better uses of its resources. By connecting human knowledge, and mining it for insights, we can pinpoint problems before they become disasters, warding off disease and shining the harsh light of data on injustice and corruption. Data is making cities smarter, watering the grass roots, and improving the way we teach.

But for every accolade, there’s a cautionary tale. It’s easy to forget that data is merely a tool, and in the wrong hands, that tool can do powerful wrong. Data erodes our privacy. It predicts us, often with unerring accuracy — and treating those predictions as fact is a new, insidious form of prejudice. And it can collect the chaff of our digital lives, harvesting a picture of us we may not want others to know.

The big data movement isn’t just about knowing more things. It’s about a fundamental shift from scarcity to abundance. Most markets are defined by scarcity — the price of diamonds, or oil, or music. But when things become so cheap they’re nearly free, a funny thing happens.

Consider the advent of steam power. Economist Stanley Jevons, in what’s known as Jevons’ Paradox, observed that as the efficiency of steam engines increased, coal consumption went up. That’s not what was supposed to happen. Jevons realized that abundance creates new ways of using something. As steam became cheap, we found new ways of using it, which created demand.

The same thing is happening with data. A report that took a month to run is now just a few taps on a tablet. An unthinkably complex analysis of competitors is now a Google search. And the global distribution of multimedia content that once required a broadcast license is now an upload.

Big data is about reducing the cost of analyzing our world. The resulting abundance is triggering entirely new ways of using that data. Visualizations, interfaces, and ubiquitous data collection are increasingly important, because they feed the machine — and the machine is hungry.

The results are controversial. Journalists rely on global access to data, but also bring a new skepticism to their work, because facts are easy to manufacture. There’s good evidence that we’ve never been as polarized, politically, as we are today — and data may be to blame. You can find evidence to support any conspiracy, expose any gaffe, or refute any position you dislike, but separating truth from mere data is a growing problem.

Perhaps the biggest threat that a data-driven world presents is an ethical one. Our social safety net is woven on uncertainty. We have welfare, insurance, and other institutions precisely because we can’t tell what’s going to happen — so we amortize that risk across shared resources. The better we are at predicting the future, the less we’ll be willing to share our fates with others. And the more those predictions look like facts, the more justice looks like thoughtcrime.

The human race underwent a huge shift when we banded together into tribes, forming culture and morals to tie us to one another. As groups, we achieved great heights, building nations, conquering challenges, and exploring the unknown. If you were one of those tribesmen, it’s unlikely you knew what was happening — it’s only in hindsight that the shift from individual to group was radical.

We’re in the middle of another, perhaps bigger, shift, one that’s taking us from physical beings to digital/physical hybrids. We’re colonizing an online world, and just as our ancestors had to create new social covenants and moral guidelines to work as groups, so we have to craft new ethics, rights and laws.

Those fighting for social change have their work cut out for them, because they’re not just trying to find justice — they’re helping to rewrite the ethical and moral guidelines for a nascent, always-on, data-driven species.

Related:

• Big data is our generation’s civil rights issue, and we don’t know it

I haven’t had this kind of response to a post before (well, I’ve had responses, such as the comments to this piece for GigaOm five years ago, but they haven’t been nearly as thoughtful).

Some of the best posts have really added to the conversation. Here’s a list of those I suggest for further reading and discussion:

Nobody notices offers they don’t get

On Oxford’s Practical Ethics blog, Anders Sandberg argues that transparency and reciprocal knowledge about how data is being used will be essential. Anders captured the core of my concerns in a single paragraph, saying what I wanted to far better than I could:

… nobody notices offers they do not get. And if these absent opportunities start following certain social patterns (for example not offering them to certain races, genders or sexual preferences) they can have a deep civil rights effect

To me, this is a key issue, and it responds eloquently to some of the comments on the original post. Harry Chamberlain commented:

However, what would you say to the criticism that you are seeing lions in the darkness? In other words, the risk of abuse certainly exists, but until we see a clear case of big data enabling and fueling discrimination, how do we know there is a real threat worth fighting?

I think that this is precisely the point: you can’t see the lions in the darkness, because you’re not aware of the ways in which you’re being disadvantaged. If whites get an offer of 20% off, but minorities don’t, that’s basically a 20% price hike on minorities — but it’s just marketing, so apparently it’s okay.

Context is everything

Mary Ludloff of Patternbuilders asks, “When does someone else’s problem become ours?” Mary is a presenter at Strata, and an expert on digital privacy. She has a very pragmatic take on things. One point Mary makes is that all this analysis is about prediction — we’re taking a ton of data and making a prediction about you:

The issue with data, particularly personal data, is this: context is everything. And if you are not able to personally question me, you are guessing the context.

If we (mistakenly) predict something, and act on it, we may have wronged someone. Mary makes clear that this is thoughtcrime — arresting someone because their behavior looked like that of a terrorist, or pedophile, or thief. Firing someone because their email patterns suggested they weren’t going to make their sales quota. That’s the injustice.

This is actually about negative rights, which Wikipedia describes as:

Rights considered negative rights may include civil and political rights such as freedom of speech, private property, freedom from violent crime, freedom of worship, habeas corpus, a fair trial, freedom from slavery.

Most philosophers agree that negative rights outweigh positive ones (i.e. I have a right to fresh air more than you have a right to smoke around me.) So our negative right (to be left unaffected by your predictions) outweighs your positive one. As analytics comes closer and closer to predicting actual behavior, we need to remember the lesson of negative rights.

Big data is the new printing press

Lori Witzel compares the advent of big data to the creation of the printing press, pointing out — somewhat optimistically — that once books were plentiful, it was hard to control the spread of information. She has a good point — we’re looking at things from this side of the big data singularity:

And as the cost of Big Data and Big Data Analytics drops, I predict we’ll see a similar dispersion of technology, and similar destabilizations to societies where these technologies are deployed.

There’s a chance that we’ll democratize access to information so much that it’ll be the corporations, not the consumers, that are forced to change.

While you slept last night

TIBCO’s Chris Taylor, standing in for Kashmir Hill at Forbes, paints a dystopian picture of video-as-data, and just how much tracking we’ll face in the future:

This makes laughable the idea of an implanted chip as the way to monitor a population. We’ve implanted that chip in our phones, and in video, and in nearly every way we interact with the world. Even paranoids are right sometimes.

I had a wide-ranging chat with Chris last week. We’re sure to spend more time on this in the future.

The veil of ignorance

The idea for the original post came from a conversation I had with some civil rights activists in Atlanta a few months ago, who hadn’t thought about the subject. They (or their parents) walked with Martin Luther King, Jr. But to them big data was “just tech.” That bothered me, because unless we think of these issues in the context of society and philosophy, bad things will happen to good people.

Perhaps the best tool for thinking about these ethical issues is the Veil of Ignorance. It’s a philosophical exercise for deciding social issues that goes like this:

1. Imagine you don’t know where you will be in the society you’re creating. You could be a criminal, a monarch, a merchant, a pauper, an invalid.
2. Now design the best society you can.

Simple, right? When we’re looking at legislation for big data, this is a good place to start. We should set privacy, transparency, and use policies without knowing whether we’re ruling or oppressed, straight or gay, rich or poor.

This post originally appeared on Solve for Interesting. This version has been lightly edited. Photo: crystal ball ii by mararie, on Flickr

Strata Conference + Hadoop World — The O’Reilly Strata Conference, being held Oct. 23-25 in New York City, explores the changes brought to technology and business by big data, data science, and pervasive computing. This year, Strata has joined forces with Hadoop World.

Save 20% on registration with the code RADAR20

Related:

• Big data is our generation’s civil rights issue, and we don’t know it
• There’s no such thing as big data

• Everything is on the Internet.
• The Internet has a lot of data.
• Therefore, everything is big data.

When you have a hammer, everything looks like a nail. When you have a Hadoop deployment, everything looks like big data. And if you’re trying to cloak your company in the mantle of a burgeoning industry, big data will do just fine. But seeing big data everywhere is a sure way to hasten the inevitable fall from the peak of high expectations to the trough of disillusionment.

We saw this with cloud computing. From early idealists saying everything would live in a magical, limitless, free data center to today’s pragmatism about virtualization and infrastructure, we soon took off our rose-colored glasses and put on welding goggles so we could actually build stuff.

So where will big data go to grow up?

Once we get over ourselves and start rolling up our sleeves, I think big data will fall into three major buckets: Enterprise BI, Civil Engineering, and Customer Relationship Optimization. This is where we’ll see most IT spending, most government oversight, and most early adoption in the next few years.

Enterprise BI 2.0

For decades, analysts have relied on business intelligence (BI) products like Hyperion, Microstrategy and Cognos to crunch large amounts of information and generate reports. Data warehouses and BI tools are great at answering the same question—such as “what were Mary’s sales this quarter?”—over and over again. But they’ve been less good at the exploratory, what-if, unpredictable questions that matter for planning and decision-making because that kind of fast exploration of unstructured data is traditionally hard to do and therefore expensive.

Most “legacy” BI tools are constrained in two ways:

• First, they’ve been schema-then-capture tools in which the analyst decides what to collect, then later captures that data for analysis.
• Second, they’ve typically focused on reporting what Avinash Kaushik (channeling Donald Rumsfeld) refers to as “known unknowns”—things we know we don’t know, and generate reports for.

These tools are used for reporting and operational purposes, and are usually focused on controlling costs, executing against an existing plan, and reporting on how things are going.

As my Strata co-chair Edd Dumbill pointed out when I asked for thoughts on this piece:

“The predominant functional application of big data technologies today is in ETL (Extract, Transform, and Load). I’ve heard the figure that it’s about 80% of Hadoop applications. Just the real grunt work of log file or sensor processing before loading into an analytic database like Vertica.”

The availability of cheap, fast computers and storage, as well as open source tools, have made it okay to capture first and ask questions later. That changes how we use data because it lets analysts speculate beyond the initial question that triggered the collection of data.

What’s more, the speed with which we can get results—sometimes as fast as a human can ask them—makes data easier to explore interactively. This combination of interactivity and speculation takes BI into the realm of “unknown unknowns,” the insights that can produce a competitive advantage or an out-of-the-box differentiator.

Cloud computing underwent a transition from promise to compromise. First big, public clouds wooed green-field startups. Then, a few years later, incumbent IT vendors introduced private cloud offerings. These private clouds included only a fraction of the benefits of their public cousins—but were nevertheless a sufficient blend of smoke, mirrors, and features to delay the inevitable move to public resources by a few years and appease the business. For better or worse, that’s where most IT cloud budgets are being spent today according to IDC, Gartner, and others. Big data adoption will undergo a similar cycle.

In the next few years, then, look for acquisitions and product introductions—and not a little vaporware—as BI vendors that enterprises trust bring them “big data lite”: enough innovation and disruption to satisfy the CEO’s golf buddies, but not so much that enterprise IT’s jobs are threatened. This, after all, is how change comes to big organizations.

Ultimately, we’ll see traditional “known unknowns” BI reporting living alongside big-data-powered data import and cleanup, and fast, exploratory data “unknown unknown” interactivity.

Civil Engineering

The second use of big data is in society and government. Already, data mining can be used to predict disease outbreaks, understand traffic patterns, and improve education.

Cities are facing budget crunches, infrastructure problems, and a crowding from rural citizens. Solving these problems is urgent, and cities are perfect labs for big data initiatives. Take a metropolis like New York: hackathons; open feeds of public data; and a population that generates a flood of information as it shops, commutes, gets sick, eats, and just goes about its daily life.

I think municipal data is one of the big three for several reasons: it’s a good tie breaker for partisanship, we have new interfaces everyone can understand, and we finally have a mostly-connected citizenry.

In an era of partisan bickering, hard numbers can settle the debate. So, they’re not just good government; they’re good politics. Expect to see big data applied to social issues, helping us to make funding more effective and scarce government resources more efficient (perhaps to the chagrin of some public servants and lobbyists). As this works in the world’s biggest cities, it’ll spread to smaller ones, to states, and to municipalities.

Making data accessible to citizens is possible, too: Siri and Google Now show the potential for personalized agents; Narrative Science takes complex data and turns it into words the masses can consume easily; Watson and Wolfram Alpha can give smart answers, either through curated reasoning or making smart guesses.

For the first time, we have a connected citizenry armed (for the most part) with smartphones. Nielsen estimated that smartphones would overtake feature phones in 2011, and that concentration is high in urban cores. The App Store is full of apps for bus schedules, commuters, local events, and other tools that can quickly become how governments connect with their citizens and manage their bureaucracies.

The consequence of all this, of course, is more data. Once governments go digital, their interactions with citizens can be easily instrumented and analyzed for waste or efficiency. That’s sure to provoke resistance from those who don’t like the scrutiny or accountability, but it’s a side effect of digitization: every industry that goes digital gets analyzed and optimized, whether it likes it or not.

Customer Relationship Optimization

The final home of applied big data is marketing. More specifically, it’s improving the relationship with consumers so companies can, as Sergio Zyman once said, sell them more stuff, more often, for more money, more efficiently.

The biggest data systems today are focused on web analytics, ad optimization, and the like. Many of today’s most popular architectures were weaned on ads and marketing, and have their ancestry in direct marketing plans. They’re just more focused than the comparatively blunt instruments with which direct marketers used to work.

The number of contact points in a company has multiplied significantly. Where once there was a phone number and a mailing address, today there are web pages, social media accounts, and more. Tracking users across all these channels — and turning every click, like, share, friend, or retweet into the start of a long funnel that leads, inexorably, to revenue is a big challenge. It’s also one that companies like Salesforce understand, with its investments in chat, social media monitoring, co-browsing, and more.

This is what’s lately been referred to as the “360-degree customer view” (though it’s not clear that companies will actually act on customer data if they have it, or whether doing so will become a compliance minefield). Big data is already intricately linked to online marketing, but it will branch out in two ways.

First, it’ll go from online to offline. Near-field-equipped smartphones with ambient check-in are a marketer’s wet dream, and they’re coming to pockets everywhere. It’ll be possible to track queue lengths, store traffic, and more, giving retailers fresh insights into their brick-and-mortar sales. Ultimately, companies will bring the optimization that online retail has enjoyed to an offline world as consumers become trackable.

Second, it’ll go from Wall Street (or maybe that’s Madison Avenue and Middlefield Road) to Main Street. Tools will get easier to use, and while small businesses might not have a BI platform, they’ll have a tablet or a smartphone that they can bring to their places of business. Mobile payment players like Square are already making them reconsider the checkout process. Adding portable customer intelligence to the tool suite of local companies will broaden how we use marketing tools.

Headlong into the trough

That’s my bet for the next three years, given the molasses of market confusion, vendor promises, and unrealistic expectations we’re about to contend with. Will big data change the world? Absolutely. Will it be able to defy the usual cycle of earnest adoption, crushing disappointment, and eventual rebirth all technologies must travel? Certainly not.

Related:

• What is big data?

What’s really driving so-called big data isn’t the volume of information. It turns out big data doesn’t have to be all that big. Rather, it’s about a reconsideration of the fundamental economics of analyzing data.

For decades, there’s been a fundamental tension between three attributes of databases. You can have the data fast; you can have it big; or you can have it varied. The catch is, you can’t have all three at once.

I’d first heard this as the “three V’s of data”: Volume, Variety, and Velocity. Traditionally, getting two was easy but getting three was very, very, very expensive.

The advent of clouds, platforms like Hadoop, and the inexorable march of Moore’s Law means that now, analyzing data is trivially inexpensive. And when things become so cheap that they’re practically free, big changes happen — just look at the advent of steam power, or the copying of digital music, or the rise of home printing. Abundance replaces scarcity, and we invent new business models.

In the old, data-is-scarce model, companies had to decide what to collect first, and then collect it. A traditional enterprise data warehouse might have tracked sales of widgets by color, region, and size. This act of deciding what to store and how to store it is called designing the schema, and in many ways, it’s the moment where someone decides what the data is about. It’s the instant of context.

That needs repeating:

You decide what data is about the moment you define its schema.

With the new, data-is-abundant model, we collect first and ask questions later. The schema comes after the collection. Indeed, big data success stories like Splunk, Palantir, and others are prized because of their ability to make sense of content well after it’s been collected — sometimes called a schema-less query. This means we collect information long before we decide what it’s for.

And this is a dangerous thing.

When bank managers tried to restrict loans to residents of certain areas (known as redlining) Congress stepped in to stop it (with the Fair Housing Act of 1968). They were able to legislate against discrimination, making it illegal to change loan policy based on someone’s race.

Home Owners’ Loan Corporation map showing redlining of “hazardous” districts in 1936.

“Personalization” is another word for discrimination. We’re not discriminating if we tailor things to you based on what we know about you — right? That’s just better service.

In one case, American Express used purchase history to adjust credit limits based on where a customer shopped, despite his excellent credit limit:

Johnson says his jaw dropped when he read one of the reasons American Express gave for lowering his credit limit: “Other customers who have used their card at establishments where you recently shopped have a poor repayment history with American Express.”

We’re seeing the start of this slippery slope everywhere from tailored credit-card limits like this one to car insurance based on driver profiles. In this regard, big data is a civil rights issue, but it’s one that society in general is ill-equipped to deal with.

We’re great at using taste to predict things about people. OKcupid’s 2010 blog post “The Real Stuff White People Like” showed just how easily we can use information to guess at race. It’s a real eye-opener (and the guys who wrote it didn’t include everything they learned — some of it was a bit too controversial). They simply looked at the words one group used which others didn’t often use. The result was a list of “trigger” words for a particular race or gender.

Now run this backwards. If I know you like these things, or see you mention them in blog posts, on Facebook, or in tweets, then there’s a good chance I know your gender and your race, and maybe even your religion and your sexual orientation. And that I can personalize my marketing efforts towards you.

That makes it a civil rights issue.

If I collect information on the music you listen to, you might assume I will use that data in order to suggest new songs, or share it with your friends. But instead, I could use it to guess at your racial background. And then I could use that data to deny you a loan.

Want another example? Check out Private Data In Public Ways, something I wrote a few months ago after seeing a talk at Big Data London, which discusses how publicly available last name information can be used to generate racial boundary maps:

Screen from the Mapping London project.

This TED talk by Malte Spitz does a great job of explaining the challenges of tracking citizens today, and he speculates about whether the Berlin Wall would ever have come down if the Stasi had access to phone records in the way today’s governments do.

So how do we regulate the way data is used?

The only way to deal with this properly is to somehow link what the data is with how it can be used. I might, for example, say that my musical tastes should be used for song recommendation, but not for banking decisions.

Tying data to permissions can be done through encryption, which is slow, riddled with DRM, burdensome, hard to implement, and bad for innovation. Or it can be done through legislation, which has about as much chance of success as regulating spam: it feels great, but it’s damned hard to enforce.

There are brilliant examples of how a quantified society can improve the way we live, love, work, and play. Big data helps detect disease outbreaks, improve how students learn, reveal political partisanship, and save hundreds of millions of dollars for commuters — to pick just four examples. These are benefits we simply can’t ignore as we try to survive on a planet bursting with people and shaken by climate and energy crises.

But governments need to balance reliance on data with checks and balances about how this reliance erodes privacy and creates civil and moral issues we haven’t thought through. It’s something that most of the electorate isn’t thinking about, and yet it affects every purchase they make.

This should be fun.

This post originally appeared on Solve for Interesting. This version has been lightly edited.

Strata Conference + Hadoop World — The O’Reilly Strata Conference, being held Oct. 23-25 in New York City, explores the changes brought to technology and business by big data, data science, and pervasive computing. This year, Strata has joined forces with Hadoop World.

Save 20% on registration with the code RADAR20

Related:

• Private Data In Public Ways
• Cooking the data
• There’s no such thing as big data

This time, we focused on how businesses are going to embrace big data, and where the challenges lie. It was a perfect opportunity to survey the attendees and get a glimpse into enterprise adoption of big data. Out of the roughly 350 attendees, approximately 100 agreed to give us their feedback on a number of questions we asked. Here are the results.

Some basic facts

While the attendees worked for a mix of commercial, educational, government, and non-profit companies, the vast majority (82%) worked for a commercial, for-profit company.

Click to enlarge.

Most of the attendees’ organizations were also fairly large — more than half of them had 500 co-workers, and 22% of them had more than 10,000.

Click to enlarge.

We used this demographic information to segment and better analyze the other three questions we asked.

Big data adoption and challenges

We then asked attendees about their journey to big data. Fewer than 20% of them already have a big data solution in place — which we clarified to mean some kind of massive-scale, sharded, NoSQL, parallel data query system that may employ interactivity and machine-assisted data exploration. More than a quarter said they have no plans at this time.

Click to enlarge.

While it’s relatively early days for adoption, more than 60% of attendees said they were in the process of gathering information on big data and what it meant to them. This is a spurious result at best: we’re of course selecting an audience that wants to be an audience. Nevertheless, the volume of attendees and their feedback suggests that deployment is ramping up: if you’re a big data vendor, this is the time to be fighting for mindshare.

Click to enlarge.

When it comes to actually deploying big data, companies have plenty of challenges. The big ones seem to be:

• Data privacy and governance.
• Defining what big data actually is.
• Integrating big data with legacy systems.
• A lack of big data skills.
• The cost of tools.

Analyzing a bit further

These results might be informative, but what we really want to know is how they correlate. After all, Strata is a data conference: we’d be remiss if we didn’t crunch things a bit!

First, we wondered whether there’s a relationship between the size of a company and the kinds of problems it’s experiencing with big data.

Click to enlarge.

Our results suggest that governance and skill shortages are problems for larger companies, and that smaller businesses worry much less about data privacy and integrating legacy systems. Cost concerns come largely from mid-sized businesses.

Then we wondered whether adoption is tied to company size.

Click to enlarge.

Among our attendees, smaller firms were ahead of the game: none of the companies larger than 500 employees said they had big data in place today.

We also found that educational, government, and NGO respondents didn’t list cost as a top concern, suggesting that they may have a tolerance for open-source or home-grown approaches.

Click to enlarge.

Of course, the number of responses from these segments isn’t statistically significant, but it warrants further study, particularly for commercial offerings trying to sell outside the for-profit world.

Finally, we wondered whether the things a company worries about change as it goes from “just browsing” to “trying to build.”

Click to enlarge.

Concerns do seem to shift over the course of adoption and maturity. Early on, companies struggle to define what big data is and worry about staffing. As they get closer to implementation, their attention shifts to legacy system integration. Once they have a system, talent shortages and a variety of other, more specific concerns emerge.

While not a hard-core study — respondents weren’t randomly selected, the number of responses within some segments isn’t statistically significant, and so on — this feedback does suggest that there’s a large demand for clear information on what big data is and how it’ll change business, and that as enterprises move to adopt these technologies they’ll face integration headaches and staffing issues.

The next free Strata Online Conference will be held on January 25. We’ll be taking a look at what’s in store for the upcoming Strata Conference (Feb 28-March 1 in Santa Clara, Calif).

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

Related:

• What is big data?
• There’s no such thing as big data
• Building data science teams

The Observe, Orient, Decide, and Act (OODA) loop. Click to enlarge.

The most important part of this loop isn’t included in the OODA acronym, however. It’s the fact that it’s a loop. The results of earlier actions feed back into later, hopefully wiser, ones. Over time, the fighter “gets inside” their opponent’s loop, outsmarting and outmaneuvering them. The system learns.

Boyd’s genius was to realize that winning requires two things: being able to collect and analyze information better, and being able to act on that information faster, incorporating what’s learned into the next iteration. Today, what Boyd learned in a cockpit applies to nearly everything we do.

Data-obese, digital-fast

In our always-on lives we’re flooded with cheap, abundant information. We need to capture and analyze it well, separating digital wheat from digital chaff, identifying meaningful undercurrents while ignoring meaningless social flotsam. Clay Johnson argues that we need to go on an information diet, and makes a good case for conscious consumption. In an era of information obesity, we need to eat better. There’s a reason they call it a feed, after all.

It’s not just an overabundance of data that makes Boyd’s insights vital. In the last 20 years, much of human interaction has shifted from atoms to bits. When interactions become digital, they become instantaneous, interactive, and easily copied. It’s as easy to tell the world as to tell a friend, and a day’s shopping is reduced to a few clicks.

The move from atoms to bits reduces the coefficient of friction of entire industries to zero. Teenagers shun e-mail as too slow, opting for instant messages. The digitization of our world means that trips around the OODA loop happen faster than ever, and continue to accelerate.

We’re drowning in data. Bits are faster than atoms. Our jungle-surplus wetware can’t keep up. At least, not without Boyd’s help.
In a society where every person, tethered to their smartphone, is both a sensor and an end node, we need better ways to observe and orient, whether we’re at home or at work, solving the world’s problems or planning a play date. And we need to be constantly deciding, acting, and experimenting, feeding what we learn back into future behavior.

We’re entering a feedback economy.

The big data supply chain

Consider how a company collects, analyzes, and acts on data.

The big data supply chain. Click to enlarge.

Let’s look at these components in order.

Data collection

The first step in a data supply chain is to get the data in the first place.

Information comes in from a variety of sources, both public and private. We’re a promiscuous society online, and with the advent of low-cost data marketplaces, it’s possible to get nearly any nugget of data relatively affordably. From social network sentiment, to weather reports, to economic indicators, public information is grist for the big data mill. Alongside this, we have organization-specific data such as retail traffic, call center volumes, product recalls, or customer loyalty indicators.

The legality of collection is perhaps more restrictive than getting the data in the first place. Some data is heavily regulated — HIPAA governs healthcare, while PCI restricts financial transactions. In other cases, the act of combining data may be illegal because it generates personally identifiable information (PII). For example, courts have ruled differently on whether IP addresses aren’t PII, and the California Supreme Court ruled that zip codes are. Navigating these regulations imposes some serious constraints on what can be collected and how it can be combined.

The era of ubiquitous computing means that everyone is a potential source of data, too. A modern smartphone can sense light, sound, motion, location, nearby networks and devices, and more, making it a perfect data collector. As consumers opt into loyalty programs and install applications, they become sensors that can feed the data supply chain.

In big data, the collection is often challenging because of the sheer volume of information, or the speed with which it arrives, both of which demand new approaches and architectures.

Ingesting and cleaning

Once the data is collected, it must be ingested. In traditional business intelligence (BI) parlance, this is known as Extract, Transform, and Load (ETL): the act of putting the right information into the correct tables of a database schema and manipulating certain fields to make them easier to work with.

One of the distinguishing characteristics of big data, however, is that the data is often unstructured. That means we don’t know the inherent schema of the information before we start to analyze it. We may still transform the information — replacing an IP address with the name of a city, for example, or anonymizing certain fields with a one-way hash function — but we may hold onto the original data and only define its structure as we analyze it.

Hardware

The information we’ve ingested needs to be analyzed by people and machines. That means hardware, in the form of computing, storage, and networks. Big data doesn’t change this, but it does change how it’s used. Virtualization, for example, allows operators to spin up many machines temporarily, then destroy them once the processing is over.

Cloud computing is also a boon to big data. Paying by consumption destroys the barriers to entry that would prohibit many organizations from playing with large datasets, because there’s no up-front investment. In many ways, big data gives clouds something to do.

Platforms

Where big data is new is in the platforms and frameworks we create to crunch large amounts of information quickly. One way to speed up data analysis is to break the data into chunks that can be analyzed in parallel. Another is to build a pipeline of processing steps, each optimized for a particular task.

Big data is often about fast results, rather than simply crunching a large amount of information. That’s important for two reasons:

1. Much of the big data work going on today is related to user interfaces and the web. Suggesting what books someone will enjoy, or delivering search results, or finding the best flight, requires an answer in the time it takes a page to load. The only way to accomplish this is to spread out the task, which is one of the reasons why Google has nearly a million servers.
2. We analyze unstructured data iteratively. As we first explore a dataset, we don’t know which dimensions matter. What if we segment by age? Filter by country? Sort by purchase price? Split the results by gender? This kind of “what if” analysis is exploratory in nature, and analysts are only as productive as their ability to explore freely. Big data may be big. But if it’s not fast, it’s unintelligible.

Much of the hype around big data companies today is a result of the retooling of enterprise BI. For decades, companies have relied on structured relational databases and data warehouses — many of them can’t handle the exploration, lack of structure, speed, and massive sizes of big data applications.

Machine learning

One way to think about big data is that it’s “more data than you can go through by hand.” For much of the data we want to analyze today, we need a machine’s help.

Part of that help happens at ingestion. For example, natural language processing tries to read unstructured text and deduce what it means: Was this Twitter user happy or sad? Is this call center recording good, or was the customer angry?

Machine learning is important elsewhere in the data supply chain. When we analyze information, we’re trying to find signal within the noise, to discern patterns. Humans can’t find signal well by themselves. Just as astronomers use algorithms to scan the night’s sky for signals, then verify any promising anomalies themselves, so too can data analysts use machines to find interesting dimensions, groupings, or patterns within the data. Machines can work at a lower signal-to-noise ratio than people.

Human exploration

While machine learning is an important tool to the data analyst, there’s no substitute for human eyes and ears. Displaying the data in human-readable form is hard work, stretching the limits of multi-dimensional visualization. While most analysts work with spreadsheets or simple query languages today, that’s changing.

Creve Maples, an early advocate of better computer interaction, designs systems that take dozens of independent, data sources and displays them in navigable 3D environments, complete with sound and other cues. Maples’ studies show that when we feed an analyst data in this way, they can often find answers in minutes instead of months.

This kind of interactivity requires the speed and parallelism explained above, as well as new interfaces and multi-sensory environments that allow an analyst to work alongside the machine, immersed in the data.

Storage

Big data takes a lot of storage. In addition to the actual information in its raw form, there’s the transformed information; the virtual machines used to crunch it; the schemas and tables resulting from analysis; and the many formats that legacy tools require so they can work alongside new technology. Often, storage is a combination of cloud and on-premise storage, using traditional flat-file and relational databases alongside more recent, post-SQL storage systems.

During and after analysis, the big data supply chain needs a warehouse. Comparing year-on-year progress or changes over time means we have to keep copies of everything, along with the algorithms and queries with which we analyzed it.

Sharing and acting

All of this analysis isn’t much good if we can’t act on it. As with collection, this isn’t simply a technical matter — it involves legislation, organizational politics, and a willingness to experiment. The data might be shared openly with the world, or closely guarded.

The best companies tie big data results into everything from hiring and firing decisions, to strategic planning, to market positioning. While it’s easy to buy into big data technology, it’s far harder to shift an organization’s culture. In many ways, big data adoption isn’t a hardware retirement issue, it’s an employee retirement one.

We’ve seen similar resistance to change each time there’s a big change in information technology. Mainframes, client-server computing, packet-based networks, and the web all had their detractors. A NASA study into the failure of Ada, the first object-oriented language, concluded that proponents had over-promised, and there was a lack of a supporting ecosystem to help the new language flourish. Big data, and its close cousin, cloud computing, are likely to encounter similar obstacles.

A big data mindset is one of experimentation, of taking measured risks and assessing their impact quickly. It’s similar to the Lean Startup movement, which advocates fast, iterative learning and tight links to customers. But while a small startup can be lean because it’s nascent and close to its market, a big organization needs big data and an OODA loop to react well and iterate fast.

The big data supply chain is the organizational OODA loop. It’s the big business answer to the lean startup.

Measuring and collecting feedback

Just as John Boyd’s OODA loop is mostly about the loop, so big data is mostly about feedback. Simply analyzing information isn’t particularly useful. To work, the organization has to choose a course of action from the results, then observe what happens and use that information to collect new data or analyze things in a different way. It’s a process of continuous optimization that affects every facet of a business.

Replacing everything with data

Software is eating the world. Verticals like publishing, music, real estate and banking once had strong barriers to entry. Now they’ve been entirely disrupted by the elimination of middlemen. The last film projector rolled off the line in 2011: movies are now digital from camera to projector. The Post Office stumbles because nobody writes letters, even as Federal Express becomes the planet’s supply chain.

Companies that get themselves on a feedback footing will dominate their industries, building better things faster for less money. Those that don’t are already the walking dead, and will soon be little more than case studies and colorful anecdotes. Big data, new interfaces, and ubiquitous computing are tectonic shifts in the way we live and work.

A feedback economy

Big data, continuous optimization, and replacing everything with data pave the way for something far larger, and far more important, than simple business efficiency. They usher in a new era for humanity, with all its warts and glory. They herald the arrival of the feedback economy.

The efficiencies and optimizations that come from constant, iterative feedback will soon become the norm for businesses and governments. We’re moving beyond an information economy. Information on its own isn’t an advantage, anyway. Instead, this is the era of the feedback economy, and Boyd is, in many ways, the first feedback economist.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

Related:

• Big data goes to work
• Building data startups: Fast, big, and focused
• Big Data Now (free anthology)
• Don’t blame the information for your bad habits

Data transparency holds promise. It should, in theory, weed out corruption and level the playing field. Rather than regulating what a company can do, for example, we can regulate what it must share with the world — and then let the world deal with the consequences, whether by boycott, activism, or class-action lawsuit. It’s something the Leading Edge Forum’s Michael Nelson described as a form of digital libertarianism: pacts of transparency between businesses and consumers, or between governments and citizens. He calls it “Mutually Assured Disclosure.”

It’s certainly encouraging to think that corruption and shenanigans wither under the harsh light of data. With information out in the open, it should be easy for interested parties to review the numbers — using cheap clouds and intuitive visualizations — and spot the cheaters.

Does data really blow its own whistle?

The first problem open data advocates run into is that of getting real information. Look at Greece: 324 Athenians reported having swimming pools on their taxes. When the government used Google Maps to try and count how many there really were, they found 16,974 of them — despite efforts by citizens to camouflage their pools under green tarpaulins. So even if activists can use widely available data to create change, that data may be wrong.

One way around this is to get your own data. The barriers to data collection have vanished with the advent of social networks, ubiquitous computing, and other innovations. Just as Greek tax officials can use Google Earth to understand tax evasion, so organizations like Asthmapolis can crowdsource data — in this case, by attaching GPS receivers to asthma inhalers — and use the information to shape public policy.

Can we tell when the data is wrong?

Data in hand, it needs to be properly analyzed. That’s not as easy as it sounds.

With software development, it’s easy to see the results. If the coder’s work isn’t effective, the finished product is buggy, unusable, slow, and incompatible. On the other hand, a lazy data scientist produces wrong results that may not be obvious to anyone. Detecting fraud or error in data sets can be tough. At Strata Summit, LinkedIn’s Monica Rogati highlighted a number of common errors that analysts make when interpreting and reporting their research; as more and more people start to work with numbers, more and more make mistakes. Statistics is often counter-intuitive. (Want a good example? Try the Monty Hall problem.)

Will we know if we’ve got bad data, whether from malice, omission, or accident? It’s possible to detect fraud in some cases. Modeling datasets often reveals problems with the data, and statisticians have tricks that can help. Benford’s Law, for example, says that “for naturally occurring data, you get more ones than twos, more twos than threes, and so on, all the way down to nine.” Point the law certain kinds of datasets, and you know how likely it is that the contents are a lie.

Will we act on it?

Open data is no good unless it leads to action. Most proponents of transparency believe that change logically flows from proof. In government, at least, current public policy suggests otherwise. On critical global issues like climate science and evolution, despite overwhelming, peer-reviewed data, we’re still deadlocked on whether to teach creationism, or whether climate change is real. Don’t like the numbers you’re getting? Call them corrupt. Threaten to take away funding. If the infographic is the new stump speech, then questioning the data is the new rebuttal.

Simply having transparency doesn’t lead to change. Without effective checks and balances, and without real punishments, shining the harsh light of data won’t do anything. This makes class action lawyers and hacktivists unlikely allies: Lawsuits, social media campaigns, and boycotts are often the only way to induce change from otherwise unregulated industries.

Data transparency is an arms race. In a world of full disclosure, cooking the data is the new cooking the books. How many of today’s data scientists will become tomorrow’s forensic accountants, locked in a war with the fraudulent and the ignorant? Open data and transparency aren’t enough: we need True Data, not Big Data, as well as regulators and lawmakers willing to act on it.

One indicator of a company’s potential is how well it can scale its business independent of human intervention. This isn’t simply the ability to automate tasks or replace workers with machines; rather, it’s the ability to augment people with data and processes.

Call it the Meat to Math ratio.

First, a comparison

To explain what I mean, I’ve done some back-of-the-napkin math on six companies. Four are public, and two have impending IPOs. Of the four public ones, two are disruptors and two are the established incumbents they’re beating to a fiscal pulp.

One common way to measure a company’s productivity is the revenue per employee.

It’s not just about revenue per employee, though. As Paul Strassman said before the first dot-com bubble in 1998, we shouldn’t give industrial-age answers to information-age questions. Rather, it’s about how well a company can leverage its employees over the long term. Companies with a good meat-to-math ratio should be able to do things like:

• Automating processes at scale.
• Maintaining genuine interactions with their customers despite a high number of customers per employee.
• Finding new businesses from their own data exhaust through introspection and experimentation.

I want to look at each of these three in more detail.

Turking, then automating

I’ve spent the last year looking at a lot of new ventures, partly because of my involvement in a startup accelerator. Our accelerator uses lean startup methodologies. These are techniques for pushing the uncertainty to the front of the company’s lifespan. Rather than getting your investment and business in place before launching, a lean model is all about doing the least amount of work to accurately predict whether a particular business will succeed. Then it’s about iterating quickly to a fit between a set of product features and a target market. It’s not a perfect science, but it’s a good way to avoid losing a lot of money on a bad idea.

One lean startup trick is doing things by hand rather than wasting time programming. Consider, for example, that you’re thinking of launching a search-by-email company. Rather than coding everything, you’d read users’ emails, search for them, and respond in an email. You’d soon find out whether people wanted to search by email, without investing time in natural language parsing, email handling, and so on. You’d be “turking,” a term that refers to the Turk (from which Amazon’s people-as-a-function-call service gets its name.)

Turking takes many forms. It might mean drawing rudimentary user interfaces, then watching someone “use” them with their finger (a process called paper prototyping). Or it might mean replacing some complex function with a human (what we jokingly refer to as a Flesh-Based API). Or maybe it’s creating landing pages for applications that don’t exist, to see who signs up. One of our incubated companies didn’t code for a month. Instead, they ran surveys and did customer development until they found something people cared about.

Early on, meat is cheaper than math.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 30% on registration with the code STN11RAD

But if a company can’t make a transition to math, it will have to turk at scale. Turking at scale is another way of describing the dirty business of managing people, with all of the chaos, uncertainty, and HR headaches it entails.

There’s an old adage among investors that a change in order of magnitude means a change in leadership. If a company goes from $10M a year to $100M a year in revenue, it’s time for a new leader. Similarly, if it goes from 100 to 1,000 customers, or from 10 to 100 employees, something has to change. Those customers and employees are meat, and meat is hard to scale.

Cloud computing and virtualization are a move from atoms to bits, replacing rack-and-stack with click-and-drag, and the resulting increases in IT productivity and server-to-administrator ratios are impressive. If you move to a virtualized, properly orchestrated IT architecture, you can survive an order-of-magnitude increase without throwing meat at the problem.

Put another way, meat is how you scale atoms — the kind that make up brick and mortar. Math is how you scale bits — the kind that make up big data businesses. Businesses that can scale bits are interesting, because bits don’t have the coefficient of friction that atoms do.

Being genuine to the masses

There’s a book called “The Clustering Of America.” At one time, it was the Bible of marketing. It broke down, zip-code by zip-code, the population of the United States. It clustered people into simple and almost laughably stereotypical groups like “Blue-blood Estates,” “Dodge Diplomats,” and “Towns & Gowns.”

It was the perfect book for a “Mad Men” era, where a pithy slogan and the right timeslot could open a million wallets. In the golden age of broadcasting, obedient audiences sat down as one around the family TV to watch a show at a time of the network’s choosing.

Today, that world is a fading memory. DVRs, iTunes, and streaming have freed us from the tyranny of the o’clock. They’ve also made it easy for us to find our niche programming. Bruce Springsteen didn’t think big enough: We have 57 million channels, and everything’s on.

Traditional marketers hate this. They’re hung over, recovering from a cheap cocktail of one-to-many, broadcast media purchases. They like buying things in big chunks, aimed at homogeneous clusters.

By contrast, modern marketing is about attention and engagement. It’s about doing something interesting, and getting tailored messages to micro-markets that expect personal attention and engagement with the companies they love. Every brand has its Little Monsters, but unlike Lady Gaga, legacy businesses don’t know how to interact with them.

Former Coca-Cola CMO Sergio Zyman describes marketing as “selling more stuff to more people more often for more money more efficiently.” In other words, selling at scale. If modern, post-broadcast marketing is about being genuine, then marketers face the challenge of being genuine (meat) at scale (math).

Companies that can work this out will win. But it will take big data systems, next-generation customer relationship management, and machine learning to help augment front-line employees.

Mining your own exhaust

Netflix and Amazon have something in common beyond their destruction of incumbents: the ability to create new businesses from whole cloth while still generating revenue. Netflix managed to become the dominant paid streaming platform, using mail-based distribution to bootstrap itself. Amazon created a cloud service from what it learned about running large-scale IT infrastructure; introduced a digital reader that now outstrips book sales; and expanded from books into many other retail markets.

Blockbuster and Barnes & Noble could have done these things, but they didn’t. Netflix and Amazon used their own data exhaust to innovate. They started new races in the middle of an existing one. As data-driven companies, they create volumes of data about their own operations, then recycle this into new insights and new businesses. Another reason for their agility is that meat has inertia: it’s hard and time-consuming to hire, fire, and retrain people; it’s easy to change an algorithm.

Back to those companies

So the meat-to-math ratio is vital for several things:

• Scaling the company without adding messy atoms and the related overhead.
• Scaling marketing without becoming disconnected from markets or customers.
• Iterating into new markets and new services adjacent to a core business.

Comparing our six firms — four public, two soon to be — in this light, what does a good meat-to-math ratio mean for your business? Let’s look closely at the two impending IPOs: Groupon and Dropbox.

Groupon’s offering, initially valued at $30B, is taking a beating. It has 7,500 employees, and it’s adding them fast. Despite the hiring binge, Groupon is seeing a promising increase in word-of-mouth sales, which is a sustainable model for scaling the business, and cost of customer acquisition is a key measure of whether a company can survive.

But Groupon has to sell to two groups: consumers looking for deals, and merchants willing to offer them. Humans have to call on small businesses directly. There are other reasons for Groupon’s troubles — the company lacks sustainable barriers to entry, as shown by competitors like Google and LivingSocial — but the root of the issue is this: Groupon is throwing meat at the growth problem, when it should be throwing math at it.

Dropbox, on the other hand, has 74 employees (a number that’s also growing fast, but is a hundred times smaller than Groupon.) Its IPO is valued at $5B, and it reportedly accepted a lower valuation in order to go with the banker it wanted. Dropbox has customer acquisition built into its model, a viral-marketing scheme where users invite their friends in return for extra free storage.

Assuming that the market values math over meat, how would you expect these two companies to compare on valuation per capita?

The way we keep score, like it or not, is market capitalization or IPO valuation. Let’s compare these six companies’ IPO values by company.

Clearly, if you’re proving you can scale with math instead of meat, the market rewards you handsomely.

In a data-driven world, the true measure of any organization, from a regional government to a global conglomerate, is its meat-to-math ratio. This sounds like a cold statement, saying machines are better than people. That’s not the point here: machines are better with people, and companies that can’t augment their employees with data and tools, that cling to antiquated ideas like broadcast, and that can’t turn their data exhaust into insight and innovation, are doomed.

Showing my math

The numbers I’ve used come from a variety of sources and time periods; they should be treated as illustrative, rather than hard data. In the interest of transparency, here’s how I got the data. If you have better numbers, I’d love to hear them.

For Amazon: Amazon had $12.95B in Q410 revenues (This includes a variety of other revenues, most significantly non-media sales and computing services), and 33,700 employees, meaning a revenue per employee of $384,273. The company had a market cap of $91.8B in early August, 2011, and roughly 43,200 employees, for a value-per-employee of $2,125,694. Sources: Google Finance; Techflash, Blorge.

For Barnes & Noble: $1.91B in Q410 revenues, and 35,000 employees, meaning a revenue per employee of $54,571. The company had a market cap of $947M in early August, 2011, and roughly 30,000 employees, for a value-per-employee of $31,567. Sources: Hoovers says 30,000 in 2011, Wikipedia says there were 40,000 employees in 2008, and Zenobank says 35,000 today.

For Netflix: $444M in Q409 revenues, and 1,000 employees, meaning a revenue-per-employee of $444,000. The company had a market cap of $12.8B in early August, 2011, and 1,000 employees, for a value-per-employee of $595.92M . Sources: Home Media Magazine; Netflix’s Adrian Cockroft tells me there are roughly 1,000 salaried contractors, plus hourly workers at distribution centres. I’m being conservative and using that same number for 2009, when it was certainly smaller.

For Blockbuster: $400M in Q409 revenues. The company peaked at 60,000 employees in 2009; I’ve assumed 55,000 by Q4, meaning a revenue-per-employee of $7,273. The company is not currently trading. Sources: Home Media Magazine, USNews.

For Groupon: Q211 revenues were $878M, with roughly 7,500 employees, for a revenue-per-employee of $117,067. The company’s IPO filing was initially valued at $30B, but will likely be significantly lower; nevertheless, I’m using the original valuation. That means a value-per-employee of $4M. Sources: Groupon S-1 filing; Business Insider says Groupon had 3,000 employees in Q4 2010, and is adding headcount aggressively. SB Online says hiring costs have jumped. And the best guess on Quora puts the count at 7,500 employees.

For Dropbox: In Q211 Dropbox had $25M in revenues, and 74 employees, for a revenue per employee of $338K. The company is planning a $5B IPO, which means a value-per-employee ratio of $67.6M. Sources: There are 74 employees on Dropbox’s website (they list them all.), TechCrunch suggests that the company chose a lower valuation than they could have in order to get the right investment bank. Business Insider estimates 2011 revenues at $100M total; I used 25 percent of these.

Related:

• There’s no such thing as big data
• Building data startups: Fast, big, and focused
• Data and the human-machine connection