"Big Data" entries

Four short links: 23 September, 2015

Four short links: 23 September, 2015

Sentence Generator, Deep Neural Networks Explainer, Sports Analytics, and System Hell

  1. Skip Thought Vectors — research (with code) that produces surrounding sentences, given a sentence.
  2. A Beginner’s Guide to Deep Neural Networks (Google) — Googlers’ 20% project to explain things to people tackles machine learning.
  3. Data Analytics in Sports — O’Reilly research report (free). When it comes to processing stats, competing companies Opta and ProZone use a combination of recording technology and human analysts who tag “events” within the game (much like Vantage Sports). Opta calculates that it tags between 1,600 and 2,000 events per football game — all delivered live.
  4. On Go, Portability, and System Interfaces — No point mentioning Perl’s Configure.sh, I thought. The poor bastard will invent it soon enough.

The security infusion

Building access policies into data stores.

Safe_Rob_Pongsajapan_Flickr_350pxHadoop jobs reflect the same security demands as other programming tasks. Corporate and regulatory requirements create complex rules concerning who has access to different fields in data sets, sensitive fields must be protected from internal users as well as external threats, and multiple applications run on the same data and must treat different users with different access rights. The modern world of virtualization and containers adds security at the software level, but tears away the hardware protection formerly offered by network segments, firewalls, and DMZs.

Furthermore, security involves more than saying yes or no to a user running a Hadoop job. There are rules for archiving or backing up data on the one hand, and expiring or deleting it on the other. Audit logs are a must, both to track down possible breaches and to conform to regulation.

Best practices for managing data in these complex, sensitive environments implement the well-known principle of security by design. According to this principle, you can’t design a database or application in a totally open manner and then layer security on top if you expect it to be robust. Instead, security must be infused throughout the system and built in from the start. Defense in depth is a related principle that urges the use of many layers of security, so that an intruder breaking through one layer may be frustrated by the next. Read more…


How advanced analytics are impacting sports

The expanding role of data analytics in a trillion-dollar industry.

Download our new free report “Data Analytics in Sports: How Playing with Data Transforms the Game,” by Janine Barlow, to learn how advanced predictive analytics are impacting the world of sports.

350px_Baseball_Players_Practicing_Thomas_Eakins_1875Sports are the perfect playing field on which data scientists can play their game — there are finite structures and distinct goals. Many of the components in sports break down numerically — e.g., number of players; length of periods; and, taking a broader view, how much each player is paid.

This is why sports and data have gone hand-in-hand since the very beginning of the industry. What, after all, is baseball without baseball cards?

In a new O’Reilly report, Data Analytics in Sports: How Playing with Data Transforms the Game, we explore the role of data analytics and new technology in the sports industry. Through a series of interviews with experts at the intersection of data and sports, we break down some of the industry’s most prominent advances in the use of data analytics and explain what these advances mean for players, executives, and fans.

Read more…


Old-school DRM and new-school analytics

Piracy isn’t the threat; it’s centuries old. Music Science is the game changer.

Download our new free report “Music Science: How Data and Digital Content are Changing Music,” by Alistair Croll, to learn more about music, data, and music science.

350px_Stephan_Sedlaczek_Mozart_am_SpinettIn researching how data is changing the music industry, I came across dozens of entertaining anecdotes. One of the recurring themes was music piracy. As I wrote in my previous post on music science, industry incumbents think of piracy as a relatively new phenomenon — as one executive told me, “vinyl was great DRM.”

But the fight between protecting and copying content has gone on for a long time, and every new medium for music distribution has left someone feeling robbed. One of the first known cases of copy protection — and illegal copying — involved Mozart himself.

As a composer, Mozart’s music spread far and wide. But he was also a performer and wanted to be able to command a premium for playing in front of audiences. One way he ensured continued demand was through “flourishes,” or small additions to songs, which weren’t recorded in written music. While Mozart’s flourishes are lost to history, researchers have attempted to understand how his music might once have been played. This video shows classical pianist Christina Kobb demonstrating a 19th century technique.

Read more…


Apache Drill: Tracking its history as an open source community

A strong, open user community needs to be fostered to reveal its potential.


A strong user community is essential to releasing the full potential of an open source project, and this influence is particularly important now for the newly developed Apache Drill project. Drill is a highly scalable SQL query engine for interactive access to a wide range of big data sources and formats. Some of the ways users have an impact are an expected part of the development process: by trying the software and reporting their experiences and use cases, users in the Drill community provide valuable feedback to developers as well as raise awareness with a larger audience of what this big data tool has to offer.

This advantage was especially important with early versions of the software; users have helped development of Drill from early days by reporting bugs and praising features that they like. And now, as Drill is reaching maturity and refinement, users likely will also provide additional innovations: experimenting with Drill in their own projects, they may find new ways to use it that had not occurred to the developers.

Drill’s flexibility and extensibility lend themselves to innovation, but there’s also a natural tendency for this type of change because the big data and Hadoop landscape also are evolving quickly. In the case of Drill, we’re seeing the “unexpectedness benefit” of openness: the community gets out ahead of the leadership in use cases and technological change.

The first big Apache Drill design meeting in September 2012 in San Jose set the tone of openness and inclusion. This was an open meeting, organized by Drill co-founder Tomer Shiran and Drill mentor Ted Dunning, and sponsored by MapR Technologies through the Bay Area Apache Drill User Group. More than 60 people attended in person, and Webex connected a larger, international audience. I recall that in addition to speaker-led presentations and discussion, long strips of paper were mounted around the room for participants to write on during breaks in order to provide ideas or offer specific ways they might want to be involved. Practical steps like this surfaced good ideas immediately, and signaled openness for future ones. Read more…

Comments: 2

Build better machine learning models

A beginner's guide to evaluating your machine learning models.


Everything today is being quantified, measured, and tracked — everything is generating data, and data is powerful. Businesses are using data in a variety of ways to improve customer satisfaction. For instance, data scientists are building machine learning models to generate intelligent recommendations to users so that they spend more time on a site. Analysts can use churn analysis to predict which customers are the best targets for the next promotional campaign. The possibilities are endless.

However, there are challenges in the machine learning pipeline. Typically, you build a machine learning model on top of your data. You collect more data. You build another model. But how do you know when to stop?

When is your smart model smart enough?

Evaluation is a key step when building intelligent business applications with machine learning. It is not a one-time task, but must be integrated with the whole pipeline of developing and productionizing machine learning-enabled applications.

In a new free O’Reilly report Evaluating Machine Learning Models: A Beginner’s Guide to Key Concepts and Pitfalls, we cut through the technical jargon of machine learning, and elucidate, in simple language, the processes of evaluating machine learning models. Read more…


Big data and the music industry: Where’s the romance?

The O'Reilly Radar Podcast: Amanda Palmer on music industry survival techniques.

Subscribe to the O’Reilly Radar Podcast to track the technologies and people that will shape our world in the years to come.


In this week’s Radar Podcast episode, author and entrepreneur Alistair Croll talks with musician and performer Amanda Palmer about the current state of the music industry and how she’s navigating her way through new platforms, crowdfunding, and an ever-increasing amount of data.

Here are a few snippets from their chat:

I’ve always approached every Internet platform and every Internet tool with the suspicion that it may not last, and that actually what’s very important is that the art and the relationships I’m building are authentic enough that even if the Internet disappeared tomorrow, or even if Facebook collapsed, or Twitter collapsed, or what have you, or all of our email went down, I’m not so reliant on the Internet itself that I couldn’t somehow piece things together.

I kind of had to come to terms with the fact that the machine still feeds what people listen to, whether it’s radio, or what gets licensed to films, or what music is playing when you walk into a shop. The ability of an artist to actually really get over that mountain, if you decide not to play the game, your hands are still pretty tied.

[Zoe Keating] was saying that an album for her is, she wants to make it all at one time and make one big statement and put it out, kind of like you would put out an opera. I’m deliberately trying to detach myself from that, and just say, ‘I wrote a song. I have this Patreon. I’m just going to put it out.’ At some point, maybe I will collect everything together so the mainstream media people of the world can have an Amanda Palmer record. But maybe that format really is dying. Read more…

Comment: 1

Big data is changing the face of fashion

How the fashion industry is embracing algorithms, natural language processing, and visual search.


Download Fashioning Data: A 2015 Update, our updated free report exploring data innovations from the fashion industry.

Fashion is an industry that struggles for respect — despite its enormous size globally, it is often viewed as frivolous or unnecessary.

And it’s true — fashion can be spectacularly silly and wildly extraneous. But somewhere between the glitzy, million-dollar runway shows and the ever-shifting hemlines, a very big business can be found. One industry profile of the global textiles, apparel, and luxury goods market reported that fashion had total revenues of $3.05 trillion in 2011, and is projected to create $3.75 trillion in revenues in 2016.

Solutions for a unique business problem

The majority of clothing purchases are made not out of necessity, but out of a desire for self-expression and identity — two remarkably difficult things to quantify and define. Yet, established brands and startups throughout the industry are finding clever ways to use big data to turn fashion into “bits and bytes,” as much as threads and buttons.

In the newly updated O’Reilly report Fashioning Data: A 2015 Update, Data Innovations from the Fashion Industry, we explore applications of big data that carry lessons for industries of all types. Topics range from predictive algorithms to visual search — capturing structured data from photographs — to natural language processing, with specific examples from complex lifecycles and new startups; this report reveals how different companies are merging human input with machine learning. Read more…


Three best practices for building successful data pipelines

Reproducibility, consistency, and productionizability let data scientists focus on the science.

Piping_EngineeringBuilding a good data pipeline can be technically tricky. As a data scientist who has worked at Foursquare and Google, I can honestly say that one of our biggest headaches was locking down our Extract, Transform, and Load (ETL) process.

At The Data Incubator, our team has trained more than 100 talented Ph.D. data science fellows who are now data scientists at a wide range of companies, including Capital One, the New York Times, AIG, and Palantir. We commonly hear from Data Incubator alumni and hiring managers that one of their biggest challenges is also implementing their own ETL pipelines.

Drawn from their experiences and my own, I’ve identified three key areas that are often overlooked in data pipelines, and those are making your analysis:

  1. Reproducible
  2. Consistent
  3. Productionizable

While these areas alone cannot guarantee good data science, getting these three technical aspects of your data pipeline right helps ensure that your data and research results are both reliable and useful to an organization. Read more…

Comments: 6
Four short links: 15 September 2015

Four short links: 15 September 2015

Bot Bucks, Hadoop Database, Futurism Biases, and Tactile Prosthetics

  1. Ashley Madison’s Fembot Con (Gizmodo) — As documents from company e-mails now reveal, 80% of first purchases on Ashley Madison were a result of a man trying to contact a bot, or reading a message from one.
  2. Terrapin — Pinterest’s low-latency NoSQL replacement for HBase. See engineering blog post.
  3. Why Futurism Has a Cultural Blindspot (Nautilus) — As the psychologist George Lowenstein and colleagues have argued, in a phenomenon they termed “projection bias,” people “tend to exaggerate the degree to which their future tastes will resemble their current tastes.”
  4. Mind-Controlled Prosthetic Arm (Quartz) — The robotic arm is connected by wires that link up to the wearer’s motor cortex — the part of the brain that controls muscle movement — and sensory cortex, which identifies tactile sensations when you touch things. The wires from the motor cortex allow the wearer to control the motion of the robot arm, and pressure sensors in the arm that connect back into the sensory cortex give the wearer the sensation that they are touching something.