Many more companies want to highlight how they're using Apache Spark in production.
One of the trends we’re following closely at Strata is the emergence of vertical applications. As components for creating large-scale data infrastructures enter their early stages of maturation, companies are focusing on solving data problems in specific industries rather than building tools from scratch. Virtually all of these components are open source and have contributors across many companies. Organizations are also sharing best practices for building big data applications, through blog posts, white papers, and presentations at conferences like Strata.
These trends are particularly apparent in a set of technologies that originated from UC Berkeley’s AMPLab: the number of companies that are using (or plan to use) Spark in production1 has exploded over the last year. The surge in popularity of the Apache Spark ecosystem stems from the maturation of its individual open source components and the growing community of users. The tight integration of high-performance tools that address different problems and workloads, coupled with a simple programming interface (in Python, Java, Scala), make Spark one of the most popular projects in big data. The charts below show the amount of active development in Spark:
For the second year in a row, I’ve had the privilege of serving on the program committee for the Spark Summit. I’d like to highlight a few areas where Apache Spark is making inroads. I’ll focus on proposals2 from companies building applications on top of Spark.
A new mantra for your next (programming) meditation session.
You might feel fine.
Can we create more vibrant intersections?
For the past two decades, the web has been a vibrant intersection of design and programming, a place where practices from art and engineering both apply. Though I’ve spent my career on the programming side – you don’t really want to see the things I design – I’ve loved the time I’ve spent working with designers.
Much of that time was frustrating, because I was frequently stuck telling designers that no, 1990s HTML couldn’t produce page layouts like QuarkXPress. The medium was different, with its own complications. However, as designers became familiar with the web, and found new ways to apply it, the conversations became richer and richer. Front-end web development became an amazing place where designers and technicians could work (and sometimes curse) together. Read more…
A closer look at the forces causing demand
Buzzwords in the software industry arise and then die off with startling frequency. Ambiguous terms such as “growth hacker”, “sales engineer” and “rockstar developer” trip a developer’s spidey sense that the person saying them is just handwaving. However, occasionally a new term is created to articulate a programming skill set based on demand due to changes in the software development industry.
In 2013 the search term “full stack developer” took off on Google Trends and began appearing in numerous tech startup job postings. In this term’s case there are several real trends driving developers to invest in learning and identifying as full stack developers.
The usage of the full stack developer term is driven by several larger trends in software development. Read more…
When ActiveRecord just isn’t enough
In Just Enough Arel, we explored a bit into how the Arel library transforms our Ruby code into SQL to be executed by the database. To do so, we discovered that Arel abstracts database tables and the fields therein as objects, which in turn receive messages not normally available in ActiveRecord queries. Wrapping up the article, we also looked at arguments for using Arel over falling back to SQL.
As alluded at the end of the previous article, Arel can do much more than merely provide a handful of comparison operators. In this post, we’ll look at how we can call native database functions, construct unions and intersects, and we’ll wrap things up by explicitly building joins with Arel.