On Being a Data Skeptic

"Modelers have a bigger responsibility now than ever before."

On Being a Data Skeptic CoverPeople come to data science in all sorts of ways. I happen to be someone who came via finance. Trained as a mathematician, I worked first at a hedge fund and then a financial risk software company, each for about two years, starting in June 2007 and ending in February 2011. If you look at those dates again, you’ll realize I had a front row seat for the financial crisis.

I worked on a few projects in algorithmic trading with Larry Summers at the hedge fund and was invited, along with the other quants at Shaw, to see him discuss the impending doom one evening with Alan Greenspan and Robert Rubin. It honestly kind of surprised and shocked me to see how little they seemed to know, or at least admitted to knowing, about the true situation in the markets. These guys were supposed to be the experts, after all.

After two years I left the hedge fund to go work in market risk––I wanted to be part of the solution to the huge mess that I was looking at and that I felt myself part of. I spent my first year at Riskmetrics trying to improve the credit default swap model before switching to account management. In other words, as part of my job, I was answering the phone when clients had problems with their risk reports. That’s when I started talking to the too-big-to-fail banks on the phone and realized that I could spend the rest of my life perfecting a mathematical model, but if nobody cared, it wouldn’t matter. And these guys didn’t care.

So here are two lessons learned from that: first, economic and mathematical models often fail, and second, there are pretty good reasons they fail–by failing, they’re often making someone, somewhere rich.

I left finance pretty disgusted with the whole thing, and because I needed to make money and because I’m a nerd, I pretty quickly realized I could rebrand myself a “data scientist” and get a pretty cool job, and that’s what I did. Once I started working in the field, though, I was kind of shocked by how positive everyone was about the “big data revolution” and the “power of data science.”

Not to underestimate the power of data––it’s clearly powerful! And big data has the potential to really revolutionize the way we live our lives for the better––or sometimes not. It really depends.

From my perspective, this was, in tenor if not in the details, the same stuff we’d been doing in finance for a couple of decades and that fields like advertising were slow to pick up on. And, also from my perspective, people needed to be way more careful and skeptical of their powers than they currently seem to be. Because whereas in finance we need to worry about models manipulating the market, in data science we need to worry about models manipulating people, which is in fact scarier. Modelers, if anything, have a bigger responsibility now than ever before.

That’s why I recently wrote a paper entitled “On Being a Data Skeptic.” I used as inspiration for my paper an article written in 2006 that warned people (primarily inside finance) about the dangers of relying too heavily on measurement. It was, unfortunately, a call that wasn’t heard then, and I’m hoping this call will be somewhat more compelling.

Download On Being a Data Skeptic for free.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England
Strata Rx Health Data Conference: April 23-25, 2014 | Boston, MA

tags: , , ,