An enterprise architecture solution for scale and efficiency.
Data processing in the enterprise goes very swiftly from “good enough” to “we need to be faster!” as expectations grow. The Zeta Architecture is an enterprise architecture that enables simplified business processes and defines a scalable way for increasing the speed of integrating data into the business. Following a bit of history and a description of the architecture, I’ll use Google as an example and look at the way the company deploys technologies for Gmail.
Origin story and motivation
I’ve worked on a variety of different information systems over my career, each with their own classes of challenge. The most interesting from a capacity perspective was for a company that delivers digital advertising. The biggest technical problems in that industry flow from the sheer volume of transactions that occur on a daily basis. Traffic flows in all hours of the day, but there are certainly peak periods, which means all planning must revolve around the capacity during the peak hours. This solution space isn’t altogether different than that of Amazon; they had to build their infrastructure to handle massive loads of peak traffic. Both Amazon and digital advertising, incidentally, have a Black Friday spike.
Many different architectural ideas came to my mind while I was in digital advertising. Real-time performance tracking of the advertising platform was one such thing. This was well before real-time became a hot buzzword in the technology industry. There was a point in time where this digital advertising company was “satisfied” with, or perhaps tolerated, having a two-to-three-hour delay between making changes to the system and having complete insight into the effects of the changes. After nearly a year at this company, I was finally able to get a large architectural change made to streamline log collection and management. Before the implementation started, I told everyone involved what would happen. Although this approach would enable the business to see the performance within approximately 5-10 minutes of the time a change was made, that this would not be good enough after people got a feel for what real-time could deliver. Since people didn’t have that taste in their mouths, they wouldn’t yet support going straight to real-time for this information. The implementation of this architecture was in place a few months after I departed the company for a new opportunity. The implementation worked great, and after about three months of experience with the new architecture, my former colleagues contacted me and told me they were looking to re-architect the entire solution to go to real time. Read more…
Widespread blockchain adoption requires understanding between developers and domain experts.
Editor’s note: this post is part of our investigation into the future of money. The full video compilation from our first event, Bitcoin & the Blockchain, is now available.
The vision for bitcoin and the blockchain is unabashedly optimistic, though already it is being realized. More and more technologists, venture capitalists, financial institutions, and even regulators are seeing its long-term potential to transform industries, from financial services to data management to the Internet of Things. In the medium term, there remain hurdles to overcome before blockchain technology can offer sufficiently compelling solutions for the complex financial and technological world we live in, but there is progress to date — and it’s promising.
Blockchain-based remittance vehicles offered by Coins.ph, BitPagos, and BitPesa, though early stage, aim to take a chunk of the $450 billion remittance industry by offering speedier, more efficient, and cheaper alternatives to traditional solutions. BitPay offers bitcoin/fiat payment processing for merchants as well as bank integration. Increasingly, private investors are diversifying their portfolios by purchasing bitcoin alongside traditional assets. Most recently, Coinbase even received funding from a group of blue-chip investors, including the New York Stock Exchange, and launched its own exchange, signaling both greater acceptance by the financial services industry as well as confidence in its future value. Ripple Labs has taken a very different approach with its protocol, permitting the decentralized transmission of practically any currency type — cryptographic or fiat — like an SMTP for money, and circumventing traditional payment networks. And to this end, it’s already inked agreements with Cross River Bank (New Jersey), CBW Bank (Kansas), and Fidor Bank (Germany), with more on the horizon. Read more…
A chat with Tony Parisi on where we are with VR, where we need to go, and why we're going to get there this time.
Consumer virtual reality (VR) is in the midst of a dizzying and exhilarating upswing. A new breed of systems, pioneered by Oculus and centered on head-worn displays with breakthrough quality, are minting believers — whether investors, developers, journalists, or early-adopting consumers. Major new hardware announcements and releases are occurring on a regular basis, game studios and production houses big and small are tossing their hats into the ring, and ambitious startups are getting funded to stake out many different application domains. Is it a boom, a bubble, or the birth of a new computing platform?
Underneath this fundamental quandary, there are many basic questions that remain unresolved: Which hardware and software platforms will dominate? What input and touch feedback technologies will prove themselves? What are the design and artistic principles in this medium? What role will standards play, who will develop them, and when? The list goes on.
For many of these questions, we’ll need to wait a bit longer for answers to emerge; like smartphones in 2007, we can only speculate about, say, the user interface conventions that will emerge as designers grapple with this new paradigm. But on other issues, there is some wisdom to be gleaned. After all, VR has been around for a long time, and there are some poor souls who have been working in the mines all along. Read more…
The O'Reilly Data Show Podcast: Mikio Braun on stream processing, academic research, and training.
Mikio Braun is a machine learning researcher who also enjoys software engineering. We first met when he co-founded a real-time analytics company called streamdrill. Since then, I’ve always had great conversations with him on many topics in the data space. He gave one of the best-attended sessions at Strata + Hadoop World in Barcelona last year on some of his work at streamdrill.
I recently sat down with Braun for the latest episode of the O’Reilly Data Show Podcast, and we talked about machine learning, stream processing and analytics, his recent foray into data science training, and academia versus industry (his interests are a bit on the “applied” side, but he enjoys both).
Using VoltDB and the Lambda Architecture to locate abnormal behavior.
Subscriber Identity Module box (SIMbox) fraud is a type of telecommunications fraud where users avoid an international outbound-calls charge by redirecting the call through voice over IP to a SIM in the country where the destination is located. This is an issue we helped a client address at Wise Athena.
Taking on this type of problem requires a stream-based analysis of the Call Detail Record (CDR) logs, which are typically generated quickly. Detecting this kind of activity requires in-memory computations of streaming data. You might also need to scale horizontally.
We recently evaluated the use of VoltDB together with our cognitive analytics and machine-learning system to analyze CDRs and provide accurate and fast SIMbox fraud detection. At the beginning, we used batch processing to detect SIMbox fraud, but the response time took too long, so we switched to a technology that allows in-memory computations in order to reach the desired time constraints.
VoltDB’s in-memory distributed database provides transactions at streaming speed in a fast environment. It can support millions of small transactions per second. It also allows streaming aggregation and fast counters over incoming data. These attributes allowed us to develop a real-time analytics layer on top of VoltDB. Read more…
The O'Reilly Radar Podcast: Vitalik Buterin on bitcoin, the blockchain, Ethereum, and the future of money.
In this Radar Podcast, I chat with Vitalik Buterin, founder of Ethereum and co-founder of Bitcoin Magazine. We met at our Bitcoin & the Blockchain summit in San Francisco to talk about the disruptive potential of the bitcoin and blockchain technologies. He also outlined some of the problems he’s trying to solve with Ethereum and weighed in on how the use cases of money are going to change over the next 10 to 20 years.
Buterin told me that his father initially introduced him to bitcoin in 2011, and he wasn’t immediately interested — in fact, he outright rejected it, thinking, “It looks like it has no intrinsic value, and it’s obviously not going to work.” As he kept hearing about, he decided to investigate more and came to the realization that ultimately led him to create the Ethereum platform:
I immediately recognized that the way bitcoin works is the way that money should work. It’s exactly the correct approach, where you have: here’s the address you’re supposed to send to, here’s how much you want to send, here’s the button to send it. It’s money made for the Internet, not like the credit card approach, where you just basically give everyone the details to take as much as they want from your bank account.
On a trip to Israel, Buterin encountered projects, such as Colored Coins and Mastercoin, using blockchain technology for things other than bitcoin currency. “They were trying to let people issue their own assets,” he said. “They were trying tack features on top, tack financial contracts on top.” The protocols, Buterin noted, were overly complicated and he realized there might be a better way: “You could make it much simpler just by replacing everything with a programming language, and then if you do that, then people can write as many features as they want in the programming language after the fact.”