A brief look at data science’s past and future

In this O'Reilly Data Show Podcast: DJ Patil weighs in on a wide range of topics in data science and big data.

Back in 2008, when we were working on what became one of the first papers on big data technologies, one of our first visits was to LinkedIn’s new “data” team. Many of the members of that team went on to build interesting tools and products, and team manager DJ Patil emerged as one of the best-known data scientists. I recently sat down with Patil to talk about his new ebook (written with Hilary Mason) and other topics in data science and big data.

Subscribe to the O’Reilly Data Show Podcast

iTunes, SoundCloud, RSS

Here are a few of the topics we touched on:

Proliferation of programs for training and certifying data scientists

Patil and I are both ex-academics who learned learned “data science” in industry. In fact, up until a few years ago one acquired data science skills via “on-the-job training.” But a new job title that catches on usually leads to an explosion of programs (I was around when master’s programs in financial engineering took off). Are these programs the right way to acquire the necessary skills? Read more…

Comment

Becoming data driven

DJ Patil and Hilary Mason's Data Driven: Creating a Data Culture is about building organizations that can take advantage of data.

I’m excited to see that DJ Patil and Hilary Mason‘s new ebook Data Driven: Creating a Data Culture is now available. It’s been a lot of fun working with DJ and Hilary over the past few months.

I’m not going to summarize their work here: you should read it. It’s based on the realization that merely assembling a bunch of people who understand statistics doesn’t do the job. You end up with a group of data specialists on the margins of the organization, who don’t have the ability to do anything more than be frustrated. If you don’t develop a data culture, if people don’t understand the value of data and how it can be used to inform discussions, you can build all the dashboards and Hadoop clusters you want, but they won’t help you.

Data is a powerful tool, but it’s easy to jump on the data bandwagon and miss the benefits. Data Driven: Creating a Data Culture is about building organizations that can really take advantage of data. Is that organization yours? Read more…

Comments: 2

The data lake model is a powerhouse for invention

In this O'Reilly Radar Podcast: Edd Dumbill on the data lake, and Rajiv Maheswaran on the science of moving dots.

In a recent blog post, Edd Dumbill, VP of strategy at Silicon Valley Data Science, wrote about the phrase “data lake.” Likening it to a dream, he described a data lake as “a place with data-centered architecture, where silos are minimized, and processing happens with little friction in a scalable, distributed environment…Data itself is no longer restrained by initial schema decisions, and can be exploited more freely by the enterprise.” He explained that he called it a “dream” because “we’ve a way to go to make the vision come true” — but noted he’s optimistic the dream can be realized.

Subscribe to the O’Reilly Radar Podcast

iTunes, SoundCloud, RSS

In this Radar Podcast epidsode, O’Reilly’s Mac Slocum sits down with Dumbill to talk about the data lake, the opportunities the model presents, and the driving forces behind the concept. Read more…

Comment

Lessons from next-generation data wrangling tools

Drawing inspiration from recent advances in data preparation.

DSC_6826_4754_Flickr

One of the trends we’re following is the rise of applications that combine big data, algorithms, and efficient user interfaces. As I noted in an earlier post, our interest stems from both consumer apps as well as tools that democratize data analysis. It’s no surprise that one of the areas where “cognitive augmentation” is playing out is in data preparation and curation. Data scientists continue to spend a lot of their time on data wrangling, and the increasing number of (public and internal) data sources paves the way for tools that can increase productivity in this critical area.

At Strata + Hadoop World New York, NY, two presentations from academic spinoff start-ups — Mike Stonebraker of Tamr and Joe Hellerstein and Sean Kandel of Trifacta — focused on data preparation and curation. While data wrangling is just one component of a data science pipeline, and granted we’re still in the early days of productivity tools in data science, some of the lessons these companies have learned extend beyond data preparation.

Scalability ~ data variety and size

Not only are enterprises faced with many data stores and spreadsheets, data scientists have many more (public and internal) data sources they want to incorporate. The absence of a global data model means integrating data silos, and data sources requires tools for consolidating schemas.

Random samples are great for working through the initial phases, particularly while you’re still familiarizing yourself with a new data set. Trifacta lets users work with samples while they’re developing data wrangling “scripts” that can be used on full data sets.
Read more…

Comments: 2

Top keynotes at Strata Conference and Strata + Hadoop World 2014

From data privacy to real-world problem solving, O’Reilly’s data editors highlight the best of the best talks from 2014.

2014 was a year of tremendous growth in the field of data, as it was as well for Strata and Strata + Hadoop World, O’Reilly’s and Cloudera’s series of data conferences. At Strata, keynotes, individual sessions, and tracks like Hardcore Data Science, Hadoop and Beyond, Data-Driven Business Day, and Design & Interfaces, among others, explore the cutting-edge aspects of how to gather, store, wrangle, analyze, visualize, and make decisions with the vast amounts of data on our hands today. Looking back on the past year of Strata, the O’Reilly data editors chose our top keynotes from Strata Santa Clara, Strata Barcelona, and Strata + Hadoop World NYC.

It was tough to winnow the list down from an exceptional set of keynotes. Visit the O’Reilly YouTube channel for a larger set of 2014 keynotes, or Safari for videos of the keynotes and many of the conference sessions.

Best of the best

  • Julia Angwin reframes the issue of data privacy as justice, due process, and human rights (and her account of trying to buy better privacy goods and services is both instructive and funny).

  • Read more…

Comment: 1

The promise and problems of big data

A look at the social and moral implications of living in a deeply connected, analyzed, and informed world.

Editor’s note: this is an excerpt from our new report Data: Emerging Trends and Technologies, by Alistair Croll. You can download the free report here.

We’ll now look at both the light and the shadows of this new dawn, the social and moral implications of living in a deeply connected, analyzed, and informed world. This is both the promise and the peril of big data in an age of widespread sensors, fast networks, and distributed computing.

Solving the big problems

The planet’s systems are under strain from a burgeoning population. Scientists warn of rising tides, droughts, ocean acidity, and accelerating extinction. Medication-resistant diseases, outbreaks fueled by globalization, and myriad other semi-apocalyptic Horsemen ride across the horizon.

Can data fix these problems? Can we extend agriculture with data? Find new cures? Track the spread of disease? Understand weather and marine patterns? General Electric’s Bill Ruh says that while the company will continue to innovate in materials sciences, the place where it will see real gains is in analytics.

It’s often been said that there’s nothing new about big data. The “iron triangle” of Volume, Velocity, and Variety that Doug Laney coined in 2001 has been a constraint on all data since the first database. Basically, you could have any two you want fairly affordably. Consider:

  • A coin-sorting machine sorts a large volume of coins rapidly, but assumes a small variety of coins. It wouldn’t work well if there were hundreds of coin types.
  • A public library, organized by the Dewey Decimal System, has a wide variety of books and topics, and a large volume of those books — but stacking and retrieving the books happens at a slow velocity.

What’s new about big data is that the cost of getting all three Vs has become so cheap it’s almost not worth billing for. A Google search happens with great alacrity, combs the sum of online knowledge, and retrieves a huge variety of content types. Read more…

Comment