A growing number of applications are being built with Spark

Many more companies want to highlight how they're using Apache Spark in production.

One of the trends we’re following closely at Strata is the emergence of vertical applications. As components for creating large-scale data infrastructures enter their early stages of maturation, companies are focusing on solving data problems in specific industries rather than building tools from scratch. Virtually all of these components are open source and have contributors across many companies. Organizations are also sharing best practices for building big data applications, through blog posts, white papers, and presentations at conferences like Strata.

These trends are particularly apparent in a set of technologies that originated from UC Berkeley’s AMPLab: the number of companies that are using (or plan to use) Spark in production1 has exploded over the last year. The surge in popularity of the Apache Spark ecosystem stems from the maturation of its individual open source components and the growing community of users. The tight integration of high-performance tools that address different problems and workloads, coupled with a simple programming interface (in Python, Java, Scala), make Spark one of the most popular projects in big data. The charts below show the amount of active development in Spark:

Apache Spark contributions

For the second year in a row, I’ve had the privilege of serving on the program committee for the Spark Summit. I’d like to highlight a few areas where Apache Spark is making inroads. I’ll focus on proposals2 from companies building applications on top of Spark.

Read more…


What it really means when people say “Everything in JavaScript is an object”

A new mantra for your next (programming) meditation session.

When you begin programming with JavaScript you might run across books, tutorials, and people who say “Everything in JavaScript is an object.” While it’s not 100% true (not *everything* is an object), it is *mostly* true. And sometimes this can be a bit surprising.

For instance, to most people functions and objects look and act completely different. And in many languages, functions and objects *are* completely different. However, in JavaScript, a function is an object. This can take a bit of concentrated attention to get your head around, but it’s an important concept because it’s the secret behind another big topic in JavaScript: functions as first class values.

Read more…


It’s the end of the web as we knew it

You might feel fine.

For the past 15 years, Google has enforced the classic “HTML as foundation” architecture at the heart of the Web. Content creators and the developers who support them had to present content and link information as part of their pages’ HTML if they wanted Google’s spidering bots to see them. Google effectively punished developers who made links or content available only through JavaScript (or images, or CSS), giving them low or non-existent search results.

Google did this to keep their processing simple, not because of a deep fondness for HTML. Even as Google’s bots stuck to a simple diet of HTML, other parts of Google were developing JavaScript-centric approaches, like AngularJS: a “Superheroic JavaScript MVW Framework” that “is what HTML would have been, had it been designed for building web-apps.”

Angular is far from alone. As JavaScript grew, more and more programmers wanted to build their apps as programs, not as pages. Or, as Jen Simmons summarized it at Fluent, “Dang that stupid HTML, I’m just going to load an empty page… then I’ll run the real program, I’ll run the JavaScript.” Read more…


Applying design values to programming

Can we create more vibrant intersections?

Design by connection by Dave Gray, on FlickrFor the past two decades, the web has been a vibrant intersection of design and programming, a place where practices from art and engineering both apply. Though I’ve spent my career on the programming side – you don’t really want to see the things I design – I’ve loved the time I’ve spent working with designers.

Much of that time was frustrating, because I was frequently stuck telling designers that no, 1990s HTML couldn’t produce page layouts like QuarkXPress. The medium was different, with its own complications. However, as designers became familiar with the web, and found new ways to apply it, the conversations became richer and richer. Front-end web development became an amazing place where designers and technicians could work (and sometimes curse) together. Read more…


Driving demand for full-stack developers

A closer look at the forces causing demand

Buzzwords in the software industry arise and then die off with startling frequency. Ambiguous terms such as “growth hacker”, “sales engineer” and “rockstar developer” trip a developer’s spidey sense that the person saying them is just handwaving. However, occasionally a new term is created to articulate a programming skill set based on demand due to changes in the software development industry.

In 2013 the search term “full stack developer” took off on Google Trends and began appearing in numerous tech startup job postings. In this term’s case there are several real trends driving developers to invest in learning and identifying as full stack developers.


The usage of the full stack developer term is driven by several larger trends in software development. Read more…


More than enough Arel

When ActiveRecord just isn’t enough

In Just Enough Arel, we explored a bit into how the Arel library transforms our Ruby code into SQL to be executed by the database. To do so, we discovered that Arel abstracts database tables and the fields therein as objects, which in turn receive messages not normally available in ActiveRecord queries. Wrapping up the article, we also looked at arguments for using Arel over falling back to SQL.

As alluded at the end of the previous article, Arel can do much more than merely provide a handful of comparison operators. In this post, we’ll look at how we can call native database functions, construct unions and intersects, and we’ll wrap things up by explicitly building joins with Arel.

Read more…