Time-series and big data

Over the last six months I’ve been encountering more data scientists (outside of finance) who work with massive amounts of time-series data. The rise of unstructured data has been widely reported, the growing importance of time-series much less so. Sources include data from consumer devices (gesture recognition & user interface design), sensors (apps for “self-tracking”), machines (systems in data centers), and health care. In fact some research hospitals have troves of EEG and ECG readings that translate to time-series data collections with billions (even trillions) of points.

Search and machine-learning at scale

Before doing anything else, one has to be able to run queries at scale. Last year I wrote about a team of researchers at UC Riverside who took an existing search algorithm (dynamic time-warping ²) and got it to scale to time-series with trillions of points. There are many potential applications of their research, one I highlighted is from health care:

… a doctor who needs to search through EEG data (with hundreds of billions of points), for a “prototypical epileptic spike”, where the input query is a time-series snippet with thousands of points.

As the size of data grows, the UCR dynamic time-warping algorithm takes time to finish (it takes a few hours for time-series with trillions of points). In general (academic) researchers who’ve spent weeks or months collecting data are fine waiting a few hours for a pattern recognition algorithm to finish. But users who come from different backgrounds (e.g. web companies) may not be as patient. Fortunately “search” is an active research area and faster (distributed) pattern recognition systems will likely emerge soon.

Once you scale up search, other interesting problems can be tackled. The UCR team is using their dynamic time-warping algorithm in tasks like classification, clustering, and motif ³ discovery. Other teams are investigating techniques from signal-processing, pattern recognition, and trajectory tracking.

Some data management tools that target time-series

One of the more popular sessions at last year’s HBase Conference was on OpenTSDB, a distributed, time series database built on top of HBase. It’s used to store and serve time series metrics, and comes with tools (based on GNUPlot) for charting. Originally named OpenTSDB2, KairosDB was written primarily for Cassandra (but also works with HBase). OpenTSDB emphasizes tools for readying data for charts (interpolating to fill in missing values), KairosDB distinguishes between data and the presentation of data.

Startup TempoDB offers a reasonably priced, cloud-based service for storing, retrieving, and visualizing time-series data. Still a work in progress SciDB is an open source database project, designed specifically for data intensive science problems. The designers of the system plan to make time-series analysis easy to express within SciDB.

(3) Motifs are similar subsequences of a long time series; shapelets are time series primitives that can be used to speed up automatic classification (by reducing the number of “features”).

This post was originally published on strata.oreilly.com.

We wanted to give you a brief update on what we’ve learned so far from our series of interviews with players and practitioners in the in-memory data management space. A few preliminary themes have emerged, some expected, others surprising.

Performance improves as you put data as close to the computation as possible. We talked to people in systems, data management, web applications, and scientific computing who have embraced this concept. Some solutions go to the the lowest level of hardware (L1, L2 cache), The next generation SSDs will have latency performance closer to main memory, potentially blurring the distinction between storage and memory. For performance and power consumption considerations we can imagine a future where the primary way systems are sized will be based on the amount of non-volatile memory^* deployed.

Putting data in-memory does not negate the importance of distributed computing environments. Data size and the ability to leverage parallel environments are frequently cited reasons. The same characteristics that make the distributed environments compelling also apply to in-memory systems: fault-tolerance and parallelism for performance. An additional consideration is the ability to gracefully spillover to disk when main is memory full.

There is no general purpose solution that can deliver optimal performance for all workloads. The drive for low latency requires different strategies depending on write or read intensity, fault-tolerance, and consistency. Database vendors we talked with have different approaches for transactional and analytic workloads, in some cases integrating in-memory into existing or new products. People who specialize in write-intensive systems identify hot data (i.e., frequently accessed) and put those in-memory.

Hadoop has emerged as an ingestion layer and the place to store data you might use. The next layer identifies and extracts high-value data that can be stored in-memory for low-latency interactive queries. Due to resource constraints of main memory, using columnar stores to compress data becomes important to speed I/O and store more in a limited space.

While it may be difficult to make in-memory systems completely transparent, the people we talked with emphasized programming interfaces that are as simple as possible.

Our conversations to date have revealed a wide range of solutions and strategies. We remain excited about the topic, and we’re continuing our investigation. If you haven’t yet, feel free to reach out to us on Twitter (Ben is @bigdata and Roger is @rogerm) or leave a comment on this post.

* By non-volatile memory we mean the next-generation SSDs. In the rest of the post “memory” refers to traditional volatile main memory.

Related:

• Need speed for big data? Think in-memory data management

In a forthcoming report we will highlight technologies and solutions that take advantage of the decline in prices of RAM, the popularity of distributed and cloud computing systems, and the need for faster queries on large, distributed data stores. Established technology companies have had interesting offerings, but what initially caught our attention were open source projects that started gaining traction last year.

An example we frequently hear about is the demand for tools that support interactive query performance. Faster query response times translate to more engaged and productive analysts, and real-time reports. Over the past two years several in-memory solutions emerged to deliver 5X-100X faster response times. A recent paper from Microsoft Research noted that even in this era of big data and Hadoop, many MapReduce jobs fit in the memory of a single server. To scale to extremely large datasets several new systems use a combination of distributed computing (in-memory grids), compression, and (columnar) storage technologies.

Another interesting aspect of in-memory technologies is that they seem to be everywhere these days. We’re looking at tools aimed at analysts (Tableau, Qlikview, Tibco Spotfire, Platfora), databases that target specific workloads or data types (VoltDB, SAP HANA, Hekaton, Redis, Druid, Kognitio, and Yarcdata), frameworks for analytics (Spark/Shark, GraphLab, GridGain, Asterix/Hyracks), and the data center (RAMCloud, memory Iocality).

We’ll be talking to companies and hackers to get a sense of how in-memory solutions fit into their planning. Along these lines, we would love to hear what you think about the rise of these technologies, as well as applications, companies and projects we should look at. Feel free to reach out to us on Twitter (Ben is @bigdata and Roger is @rogerm) or leave a comment on this post.

Hadoop integration: Spark can work with files stored in HDFS, an important feature given the amount of investment in the Hadoop Ecosystem. Getting Spark to work with MapR is straightforward.

The Spark interactive Shell: Spark is written in Scala, and has it’s own version of the Scala interpreter. I find this extremely convenient for testing short snippets of code.

The Spark Analytic Suite:

(Figure courtesy of Matei Zaharia)

Spark comes with tools for interactive query analysis (Shark), large-scale graph processing and analysis (Bagel), and real-time analysis (Spark Streaming). Rather than having to mix and match a set of tools (e.g., Hive, Hadoop, Mahout, S4/Storm), you only have to learn one programming paradigm. For SQL enthusiasts, the added bonus is that Shark tends to run faster than Hive. If you want to run Spark in the cloud, there are a set of EC2 scripts available.

Resilient Distributed Data sets (RDD’s):
RDD’s are distributed objects that can be cached in-memory, across a cluster of compute nodes. They are the fundamental data objects used in Spark. The crucial thing is that fault-tolerance is built-in: RDD’s are automatically rebuilt if something goes wrong. If you need to test something out, RDD’s can even be used interactively from the Spark interactive shell.

Distributed Operators:
Aside from Map and Reduce, there are many other operators one can use on RDD’s. Once I familiarized myself with how they work, I began converting a few basic machine-learning and data processing algorithms into this framework.

Once you get past the learning curve … iterative programs
It takes some effort to become productive in anything, Spark is no exception. I was a complete Scala newbie so I first had to get comfortable with a new language (apparently, they like underscores – see here, here, and here). Beyond Scala one can use Shark (“SQL” on Spark), and relatively new Java and Python API’s.

You can use the examples that come with Spark to get started, but I found the essential thing is to get comfortable with the built-in distributed operators. Once I learned RDD’s and the operators, I started writing iterative programs to implement a few machine-learning and data processing algorithms. (Since Spark distributes & caches data in-memory, you can write pretty fast machine-learning programs on massive data sets.)

It’s already used in production
Is anyone really using Spark? While the list of companies is still small, judging from the size of the SF Spark Meetup and Amp Camp, I expect many more companies to start deploying Spark. (If you’re in the SF Bay Area, we are starting a new Distributed Data Processing Meetup with Airbnb, and Spark is one one of the topics we’ll cover.)

Update (8/23/2012): Here’s another important reason to like Spark – at 14,000 lines of code it’s much simpler than other software used for Big Data.

The Spark codebase is small, extensible, and hackable.
Matei’s last presentation at Amp Camp included the diagram below (LOC = lines of code):

(Figure courtesy of Matei Zaharia)

I. Percentage share of active users (weekly): note the steady rise in the share of users from Asia.

II. Market Penetration: Less than 3% penetration in Facebook’s high-growth regions in Asia & Africa.

III. Percent Share of each Age Group (within each Region): Relative to the U.S. (33%), the share of users ages 18-25 continues to be higher in Asia (44%), Africa (41%), and the Middle East / N. Africa (39%).

Below is the breakdown by most popular occupations over the last three months:

[For a similar breakdown by location (most popular metro areas), click HERE.]

(*) Using data from a partnership with SimplyHired, we maintain a data warehouse that includes most U.S. online job postings dating back to late 2005. Since there are no standard data formats for job postings across employment sites, algorithms are used to detect duplicate job postings, companies, occupations, and metro areas. The algorithms are far from perfect, so the above results are at best extremely rough estimates.

(**) As a benchmark, the total number of job postings in our entire data warehouse of jobs grew 68% from Aug/Oct 2009 to Aug/Oct 2010.

The Windows Marketplace for Mobile now has about 1,400 apps spread across 16 categories. In this short post I’ll provide some basic statistics* and compare it with the grandaddy of app stores – the U.S. iTunes store.

First let’s look at the distribution of apps across categories. Like the iPhone and Android platforms, Windows Phone 6.x are rich in game apps. Given that there are far fewer Windows Phone apps, it may take some time before we see the variety of categories found in iTunes. There are large iPhone categories (medical**, education, sports … ) that aren’t part of the taxonomy for Windows Marketplace for Mobile.

More than 90% of the 280,000+ iTunes apps aren’t free, compared to 78% of apps available on Windows Marketplace for Mobile. Below are the share of free/paid apps across the different categories.

At least for now, Windows Phone 6.x apps are pricier than iPhone apps. The mean price of a paid iPhone app is $3.43, compared to $6.16 for paid apps available on Windows Marketplace for Mobile. Welcome news for the many developers gearing up to produce apps for Windows Phone 7!

(*) Data for this post: U.S. iTunes store through 10/31/2010, limited to iPhone apps; Windows Marketplace for Mobile through 11/3/2010.

(**) The Medical category was added several months after the launch of the iTunes app store.

One company that recently caught my eye is Helsinki (and SF) based Microtask. Founded by Computer Graphics (CG) and Computer Vision (CV) veterans, Microtask has chosen to focus in a few areas where CG and CV are relevant. Aside from speech transcription, they currently provide form-processing (digitizing hand-written forms) and archive digitization services, and have plans to expand into image categorization and video indexing in the near future. By initially focusing on a few specific tasks, Microtask is able to refine its platform while simultaneously leveraging prior skills in areas such as optical character recognition.

A few things about the Microtask platform are worth highlighting. In order to protect the intellectual property of its customers, Microtask never sends complex tasks to the same service provider. Rather tasks get broken up into pieces and scattered across multiple providers. This is fairly easy to do for the types of digitization services they offer. Customers who are wary of sending data to outside servers, can run Microtask’s software on their own servers. (Ordinarily customers use Microtask through a set of API’s.) Finally, Microtask can guarantee quality and delivery time because it has longterm relationships with (labor) service providers. Microtask contracts out with call-centers throughout the world, and tasks* are performed by workers in-between service calls.

Using call-center workers is novel but crowdsourcing seems increasingly tied to social gaming and virtual currencies. Having come from Computer Graphics and Computer Vision, the founders of Microtask have experience and connections in the gaming industry. CEO Ville Miettinen admitted that social gaming integration is a high-priority for them over the next few years. The key is that they want Microtask to fit seamlessly into the gaming experience, they want gamers to be able to stay “in the flow of the game” while performing crowdsourcing tasks. I’m looking forward to what they and game designers come up with — a modern equivalent of Typing of the Dead?

(*) It’s useful to remember that these are simple tasks (e.g., OCR) involving the validation of outputs generated using machine-learning. Microtask uses confidence scores generated by their algorithms to rank and prioritize validation tasks.

Measured in terms of (U.S.) job postings, Amazon’s Cloud Computing platform is still larger than Google’s App Engine. What’s interesting is that the gap has closed over the past year²:

Over the past two months, the other cloud platforms were roughly one-third (Google), one-fourth (Microsoft), and one-sixth (Rackspace) the size of Amazon. During the same period last year, these platforms were much smaller: Google was one-fifth, Microsoft was one-seventh, and Rackspace one-tenth the size of Amazon.

(1) Data for this post is for U.S. online job postings through 8/21/2010 and is maintained in partnership with SimplyHired.com. We use algorithms to dedup job posts: a single job posting can contain multiple jobs and appear on multiple job sites.

(2) I counted the number of unique job posts that mention each of the cloud computing platforms.

The fraction of “hadoop” jobs posted by California companies remain high, but is definitely lower than what it was last year:

(1) Over the last three months, job posts that mention “hadoop” were inching towards 8-10% of the number of job posts that mention “mysql”.

(2) Data for this post is for U.S. online job postings through 7/31/2010 and is maintained in partnership with SimplyHired.com. We use algorithms to dedup job posts: a single job posting can contain multiple jobs and appear on multiple job sites.