Compelling large-scale data platforms originate from the world of IT Operations
I’ve been noticing that many interesting big data systems are coming out of IT operations. These are systems that go beyond the standard “capture/measure, display charts, and send alerts”. IT operations has long been a source of many interesting big data1 problems and I love that it’s beginning to attract the attention2 of many more data scientists and data engineers.
It’s not surprising that many of the interesting large-scale systems that target time-series and event data have come from ops teams: in an earlier post on time-series, several of the tools I highlighted came out of IT operations. IT operations involves monitoring many different hardware and software systems, a task that requires a variety of tools and which quickly leads to “metrics overload”. A partial list includes data captured from a wide range of application log files, network traffic, energy and power sources.
The volume of IT ops data has led to new tools like OpenTSDB and KairosDB – time series databases that leverage HBase and Cassandra. But storage, simple charts, and lookups are just the foundation of what’s needed. IT Ops track many interdependent systems, some of which might be correlated3. Not only are IT ops faced with highlighting “unknown unknowns” in their massive data sets, they often need to do so in near realtime.
OSCON 2013 Speaker Series
Automating the configuration management of your operating systems and the rollout of your applications is one of the most important things an administrator or developer can do to avoid surprises when updating services, scaling up, or recovering from failures. However, it’s often not enough. Some of the most common operations that happen in your datacenter (or cloud environment) involve large numbers of machines working together and humans to mediate those processes. While we have been able to remove a lot of human effort from configuration, there has been a lack of software able to handle these higher-level operations.
I used to work for a hosted web application company where the IT process for executing an application update involved locking six people in a room for sometimes 3-4 hours, each person pressing the right buttons at the right time. This process almost always had a glitch somewhere where someone forgot to run the right command or something wasn’t well tested beforehand. While some technical solutions were applied to handle configuration automation, nothing that could perform configuration could really accomplish that high level choreography on top as well. This is why I wrote Ansible.
Ansible is a configuration management, application deployment, and IT orchestration system. One of Ansible’s strong points is having a very simple, human readable language – it allows users very fine, precise control over what happens on what machines at what times.
To get started, create an inventory file, for instance, ~/ansible_hosts that defines what machines you are managing, and which machines are frequently organized into groups. Ansible can also pull inventory from multiple cloud sources, but an inventory file is a quick way to get started:
# add more webservers here
Now that you have defined what machines you are managing, you have to define what you are going to do on the remote machines.
Ansible calls this description of processes a “playbook,” and you don’t have to have just one, you could have different playbooks for different kinds of tasks.
Let’s look at an example for describing a rolling update process. This example is somewhat involved because it’s using haproxy, but haproxy is freely available. Ansible also includes modules for dealing with Netscalers and F5 load balancers, so this is just an example — ordinarily you would start more simply and work up to an example like this:
Velocity 2013 Speaker Series
At some point, we’ve all ended up trading horror stories over drinks with colleagues. Heads nod and shake in sympathy, and the stories get hairier as the night goes on. And while it of course feels good to get some of that dirt off your shoulder, is there a larger, better purpose to sharing war stories? I sat down with James Turnbull of Puppet Labs (@kartar) to chat about his upcoming Velocity talk about Ops mythology, and how we might be able to turn our tales of disaster into triumph.
Key highlights of our discussion include:
- Why do we share disaster stories? What is the attraction? [Discussed at 0:40]
- Stories are about shared experience and bonding with members of our community. [Discussed at 2:10]
- These horror stories are like mythological “big warnings” that help enforce social order, which isn’t always a good thing. [Discussed at 4:18]
- A preview of how his talk will be about moving away from the bad stories so people can keep telling more good stories. (Also: s’mores.) [Discussed at 7:15]
You can watch the entire interview here:
This is one of a series of posts related to the upcoming Velocity conference in Santa Clara, CA (June 18-20). We’ll be highlighting speakers in a variety of ways, from video and email interviews to posts by the speakers themselves.
Velocity 2013 Speaker Series
While automation is clearly making everyone’s lives who work in Operations much better, startup founder Kate Matsudaira (@katemats) acknowledges that “No one ever does their work in a vaccum.” You can try as much as possible to Automate All The Things, but you can’t automate trust. And trust is key to a healthy, thriving operations team (and your own professional growth, too).
In this interview, Kate discusses some of the things she’ll be talking about at Velocity next month. Key highlights include:
- The word “people” is pretty broad. What aspects of working with people should operations teams care about? [Discussed at 1:32]
- Ultimately, you depend on the people around you to help get work done, especially when you need to get funding, be it externally for a startup, or internally for an infrastructure or refactoring project. The more people trust you, the more likely that is to happen. [Discussed at 3:17]
- Cultural change takes leadership, but that leadership doesn’t have to come from the top. [Discussed at 5:00]
- You can be ridiculously technically competent, but if you can’t communicate well, it hinders your success in the long run. [Discussed at 5:40]
You can view the entire interview here:
This is one of a series of posts related to the upcoming Velocity Conference in Santa Clara, CA (June 18-20). We’ll be highlighting speakers in a variety of ways, from video and email interviews to posts by the speakers themselves.
Web ops and performance questions with Schlomo Schapiro.
A profile of web operations and performance expert Schlomo Schapiro, systems architect and open source evangelist at ImmobilienScout24.
What we mean by "operations," and how it's changed over the years.
NoOps, DevOps — no matter what you call it, operations won’t go away. Ops experts and development teams will jointly evolve to meet the challenges of delivering reliable software to customers.
Web ops and performance questions with Kate Matsudaira.
A profile of web operations and performance expert Kate Matsudaira, vice president of engineering at Decide.com.
Kate Matsudaira on the human side of performance and operations.
In this Velocity podcast, Kate Matsudaira discusses the human side of performance and operations, including how to teach people to address time, cost, quality and scope.
Shifts for sysadmins and a surprising use for Chef.
OpsCode chief community officer Jesse Robbins discusses cloud infrastructure automation and the most surprising use of Chef he's seen so far.
Carsharing boosts city governments, why complex systems fail, and what web ops teams could do with big data.
This week on O'Reilly: How Zipcar's technology is saving big money for U.S. city governments, why scalable clouds need simple parts, and pondering the possibilities of web ops and machine learning.