FEATURED STORY

The promise of Promise Theory: The O’Reilly Radar Podcast

Mark Burgess chats about Promise Theory, and Geoffrey Moore discusses a modern approach to his Crossing the Chasm theory.

Editor’s note: you can subscribe to the O’Reilly Radar Podcast through iTunes,SoundCloud, or directly through our podcast’s RSS feed.

As systems become increasingly distributed and complex, it’s more important than ever to find ways to accurately describe and analyze those systems, and to formalize intent behind processes, workflows, and collaboration.

In this podcast episode, I chat with Mark Burgess, founder and CTO of CFEngine, about the origins of Promise Theory and its connection to DevOps. Read more…

Comment

Announcing Spark Certification

A new partnership between O’Reilly and Databricks offers certification and training in Apache Spark.

Editor’s note: full disclosure — Ben is an advisor to Databricks.

spark-logoI am pleased to announce a joint program between O’Reilly and Databricks to certify Spark developers. O’Reilly has long been interested in certification, and with this inaugural program, we believe we have the right combination — an ascendant framework and a partnership with the team behind the technology. The founding team of Databricks comprises members of the UC Berkeley AMPLab team that created Spark.

The certification exam will be offered at Strata events, through Databricks’ Spark Summits, and at training workshops run by Databricks and its partner companies. A variety of O’Reilly resources will accompany the certification program, including books, training days, and videos targeted at developers and companies interested in the Apache Spark ecosystem. Read more…

Comment

One man willingly gave Google his data. See what happened next.

Google requires quid for its quo, but it offers something many don’t: user data access.

Despite some misgivings about the company’s product course and service permanence (I was an early and fanatical user of Google Wave), my relationship with Google is one of mutual symbiosis. Its “better mousetrap” approach to products and services, the width and breadth of online, mobile, and behind-the-scenes offerings saves me countless hours every week in exchange for a slice of my private life, laid bare before its algorithms and analyzed for marketing purposes.

I am writing this on a Chromebook by a lake, using Google Docs and images in Google Drive. I found my way here, through the thick underbrush along a long since forgotten former fishmonger’s trail, on Google Maps after Google Now offered me a glimpse of the place as one of the recommended local attractions.

lake

The lake I found via Google Maps and a recommendation from Google Now.

Admittedly, having my documents, my photos, my to-do lists, contacts, and much more on Google, depending on it as a research tool and mail client, map provider and domain host, is scary. And as much as I understand my dependence on Google to carry the potential for problems, the fact remains that none of those dependencies, not one shred of data, and certainly not one iota of my private life, is known to the company without my explicit, active, consent. Read more…

Comments: 26

How Flash changes the design of database storage engines

High-performing memory throws many traditional decisions overboard

supermicro_storage

Over the past decade, SSD drives (popularly known as Flash) have radically changed computing at both the consumer level — where USB sticks have effectively replaced CDs for transporting files — and the server level, where it offers a price/performance ratio radically different from both RAM and disk drives. But databases have just started to catch up during the past few years. Most still depend on internal data structures and storage management fine-tuned for spinning disks.

Citing price and performance, one author advised a wide range of database vendors to move to Flash. Certainly, a database administrator can speed up old databases just by swapping out disk drives and inserting Flash, but doing so captures just a sliver of the potential performance improvement promised by Flash. For this article, I asked several database experts — including representatives of Aerospike, Cassandra, FoundationDB, RethinkDB, and Tokutek — how Flash changes the design of storage engines for databases. The various ways these companies have responded to its promise in their database designs are instructive to readers designing applications and looking for the best storage solutions.

Read more…

Comments: 2

Building pipelines to facilitate data analysis

A new operator from the magrittr package makes it easier to use R for data analysis.

Construction_of_Cedar_River_Pipeline_1900

In every data analysis, you have to string together many tools. You need tools for data wrangling, visualisation, and modelling to understand what’s going on in your data. To use these tools effectively, you need to be able to easily flow from one tool to the next, focusing on asking and answering questions of the data, not struggling to jam the output from one function into the format needed for the next. Wouldn’t it be nice if the world worked this way! I spend a lot of my time thinking about this problem, and how to make the process of data analysis as fast, effective, and expressive as possible. Today, I want to show you a new technique that I’m particularly excited about.

R, at its heart, is a functional programming language: you do data analysis in R by composing functions. However, the problem with function composition is that a lot of it makes for hard-to-read code. For example, here’s some R code that wrangles flight delay data from New York City in 2013. What does it do? Read more…

Comment: 1

Ten years of OpenStreetMap

OSM is moving out of its awkward adolescence and into its mature, young adult phase.

OSM_logo-2Next to GPS, the most significant development in the Open Geo Data movement is OpenStreetMap (OSM), a community-driven mapping project whose goal is to create the most detailed, correct, and current open map of the world. This week, OSM celebrates its 10th birthday, which provides a convenient excuse to highlight why its achievements to-date are amazing, unusual, and promising in equal parts.

When the project was begun by Steve Coast in 2004, map data sources were few, and largely controlled by a small collection of private and governmental players. The scarcity of map data ensured that it remained both expensive and highly restrictive, and no one but the largest navigation companies could use map data. Steve changed the rules by creating a wiki-like resource of the entire globe, which everyone could use without hinderance. Read more…

Comment: 1