Key insights from DevOps, Web operations, and performance.
People from across the Web operations and performance worlds came together for the 2015 O’Reilly Velocity Conference in Amsterdam. Below, we’ve assembled notable material from the event.
The Physical Web: A bridge between the Web and physical devices
The app-for-everything approach doesn’t scale, but the Web does. Scott Jenson, project lead for Physical Web at Google, outlines a vision for the Physical Web — an open approach to design and implementation that brings Web interaction to the physical world. “Let’s take the URL bar and bring it in the future,” Jenson says.
Comparing different orchestration tools.
Most software systems evolve over time. New features are added and old ones pruned. Fluctuating user demand means an efficient system must be able to quickly scale resources up and down. Demands for near zero-downtime require automatic fail-over to pre-provisioned back-up systems, normally in a separate data centre or region.
On top of this, organizations often have multiple such systems to run, or need to run occasional tasks such as data-mining that are separate from the main system, but require significant resources or talk to the existing system.
When using multiple resources, it is important to make sure they are efficiently used — not sitting idle — but can still cope with spikes in demand. Balancing cost-effectiveness against the ability to quickly scale is difficult task that can be approached in a variety of ways.
All of this means that the running of a non-trivial system is full of administrative tasks and challenges, the complexity of which should not be underestimated. It quickly becomes impossible to look after machines on an individual level; rather than patching and updating machines one-by-one they must be treated identically. When a machine develops a problem it should be destroyed and replaced, rather than nursed back to health.
Various software tools and solutions exist to help with these challenges. Let’s focus on orchestration tools, which help make all the pieces work together, working with the cluster to start containers on appropriate hosts and connect them together. Along the way, we’ll consider scaling and automatic failover, which are important features.
Key insights from DevOps, Web operations, and performance.
People from across the Web operations and performance worlds are coming together this week for the 2015 O’Reilly Velocity Conference in New York. Below, we’ve assembled notable keynotes, interviews, and insights from the event.
Diversity and bias
Bryan Liles, who works on strategic initiatives for DigitalOcean, gave a great thought-provoking talk on bias and diversity. “If your company is creating a diversity plan and you’ve actually gone and counted people,” Liles said, “you’ve already lost.”
The DevOps Audit Defense Toolkit tries to make a case to an auditor for Continuous Deployment in a regulated environment: that developers, following a consistent, disciplined process, can safely push changes out automatically to production once the changes pass all of the reviews and automated tests and checks in the CD pipeline.
Continuous Deployment has been made famous at places like Flickr, IMVU (where Eric Ries developed the ideas for the Lean Startup method), and Facebook:
Facebook developers are encouraged to push code often and quickly. Pushes are never delayed and [are] applied directly to parts of the infrastructure. The idea is to quickly find issues and their impacts on the rest of the system and surely [fix] any bugs that would result from these frequent small changes.1
While organizations like Etsy and Wealthfront work hard to make Continuous Deployment safe, it is scary to auditors, to operations managers, and to CTOs like me who have been working in financial technology and understand the risks involved in making changes to a live, business-critical system.
Continuous Deployment requires you to shut down a running application on a server or a virtual machine, load new code, and restart. This isn’t that much of a concern for stateless web applications with pooled connections, where browser users aren’t likely to notice that they’ve been switched to a new environment in Blue-Green deployment.2 There are well-known, proven techniques and patterns for doing this that you can follow with confidence for this kind of situation.
How DevOps will help you surpass the common challenges of financial services software development.
Download a free copy of DevOps for Finance, an O’Reilly report by Jim Bird for the financial services software insider who’s heard about DevOps, but is unsure whether it represents solution or suicide.
DevOps, until recently, has been a story about unicorns. Innovative, engineering-driven online tech companies like Flickr, Etsy, Twitter, Facebook, and Google. Netflix and its Chaos Monkey. Amazon deploying thousands of changes per day.
DevOps was originally about WebOps at Internet companies working in the Cloud, and a handful of Lean Startups in Silicon Valley. It started at these companies because they had to move quickly, so they found new, simple, and collaborative ways of working that allowed them to innovate much faster and to scale much more effectively than organizations had done before.
But as the velocity of change in business continues to increase, enterprises — sometimes referred to as “horses,” in contrast to the unicorns referenced above — must also move to deliver content and features to customers just as quickly. These large organizations have started to adopt (and, along the way, adapt) DevOps ideas, practices, and tools.
This short book assumes that you have heard about DevOps and want to understand how DevOps practices like Continuous Delivery and Infrastructure as Code can be used to solve problems in financial systems at a trading firm, or a big bank or stock exchange. We’ll look at the following key ideas in DevOps, and how they fit into the world of financial systems:
- Breaking down the “wall of confusion” between development and operations, and extending Agile practices and values from development to operations
- Using automated configuration management tools like Chef, Puppet, CFEngine, or Ansible to programmatically provision and configure systems (Infrastructure as Code)
- Building Continuous Integration and Continuous Delivery (CI/CD) pipelines to automatically test and push out changes, and wiring security and compliance into these pipelines
- Using containerization and virtualization technologies like Docker and Vagrant, together with Infrastructure as Code, to create IaaS, PaaS, and SaaS clouds
- Running experiments, creating fast feedback loops, and learning from failure
To follow this book you need to understand a little about these ideas and practices. There is a lot of good stuff about DevOps out there, amid the hype. Read more…
The O'Reilly Radar Podcast: Astrid Atkinson on optimization, and Kelsey Hightower on distributed computing.
Subscribe to the O’Reilly Radar Podcast to track the technologies and people that will shape our world in the years to come.
In this week’s episode, O’Reilly’s Mac Slocum talks to Astrid Atkinson, director of software engineering at Google, about the delicate balance of managing complexity in distributed systems and her experience working on-call rotations at Google.
Here are a few snippets from their chat:
I think it’s often really hard for organizations that are scaling quickly to find time to manage complexity in their systems. That can be really a trap, because if you’re really always just focused on the next deadline or whatever, and never planning for what you’re going to live with when you’re done, then you might never find the time.
You can only optimize what you pay attention to, and so if you can’t see what your system is doing, if you can’t see whether it’s working, it’s not working.
I used to get paged awake at two in the morning. You go from zero to Google is down. That’s a lot to wake up to.