In this special-edition O’Reilly Podcast, O’Reilly’s Ben Lorica and VoltDB’s co-founder Scott Jarr discuss how VoltDB’s hybrid transaction, analytic system allows for real-time analytics and personalization of data across various industries.
Scaling transaction processing without losing the relational database
MIT’s Mike Stonebraker (VoltDB’s co-founder) wanted to scale traditional OLTP (online transaction processing) without losing performance. The project evolved and eventually commercialized as VoltDB around the time NoSQL systems introduced a paradigm shift to non-relational databases. Jarr describes how Stonebraker’s approach didn’t assume a relational database was a core issue:
To give you an old story, but it’s a good story, they took a traditional style OLTP database and they ran it in memory. What they found was that it was doing less than 10% of its effective workload in processing transactions. The rest was dealing with overhead in various forms. He said, ‘Without getting rid of any of the things that we know [are] involved in the database world — consistency, SQL, ACID transactions, relational structures, high-level query languages — let’s keep all that, but let’s see if we can make this thing go faster.’
When those [NoSQL] systems were coming out, and they were coming out very strong, it was around the same time we were coming out with VoltDB. People were asking questions, ‘Well you’re consistent and they’re not.’ Or, ‘You’re relational and they’re not.’ I think that really lost the true meaning of what the differences were … [let’s] not get mired in the details … let’s look at the workloads that people are trying to accomplish.
Automated decision-making
Analytics that used to take weeks can now take hours or minutes. VoltDB allows you to contextualize data by joining real-time streams to static data in a single system. For example, tying transactions to a table of customer demographic data (even if the demographics are still dynamic). Jarr describes how market-need introduced the possibility of real-time decision-making:
Customers were telling us analytics are also important in the transaction. I mean that very literally — ‘I’m going to use these real-time analytics … as part of my decision-making process in a transaction.’
Our theory and our belief … is that once you start to get to this level [where] the analytics are real time, that means the decision needs to be real time, which means the decision needs to be automated. Now you’re talking about this convergence of streaming real-time analytics and the operational database.
Many of these streaming analytics-alone solutions are nice if you’re just looking to do real-time dashboards. The moment that you’re looking to actually react as a data transaction in real time with those analytics, now you’re talking about a database problem.
The value of real-time analysis (IoT strikes again)
Real-time analytics isn’t new, but it was limited to industries (finance, telco) that could afford to build the massive systems it required. Jarr describes how VoltDB enables personalization of data across new industries:
I hate to even say the words, because it’s so buzz-wordy right now, but it’s the IoT.
The UK is rolling out 53 million smart meters over the next 10 years … VoltDB is the front end, and it’s ingesting all those meter readings and it’s doing security, data quality, some trending analysis in the low-level real-time analytics area. It’s also then handing it off … to something that looks very much like a data lake in the back end. IoT-type things really do become transactions in the front.
Mobile is a really good [use case] as well. These are the major global telco-type customers that have realized that there’s a tremendous amount of valuable data that’s coming off of these mobile devices and they should be able to do stuff with that — whether those things are for contextual marketing applications, or they’re for enforcing bill shock and regulatory requirements around telco interactions, or they’re just about new platforms that they’re building out.
Personalization is a phenomenal application of real-time transactions. One day I had a conversation with somebody about ad placement, then I had a conversation with somebody in the financial services industry. Then I had a conversation with somebody in the biomedical industry, and what I realized is all three use cases in those three very different industries were all about personalization of information. That is a very real-time operational interaction that requires the same things that we’re talking about. It’s streaming analytics and a transaction at the same time.
As we move more toward sensors everywhere, mobile everywhere, Internet everywhere, everything connected, we start to see these applications … touching every industry.
You can listen to the podcast in the player above or on SoundCloud.
This podcast is a collaboration between O’Reilly and VoltDB. See our statement of editorial independence.