"devops" entries

Chaos Monkey for systems of people: The ultimate performance hack

The O'Reilly Radar Podcast: Alois Reitbauer on performance hacking, DevOps applications, and fostering a culture of respect.


Subscribe to the O’Reilly Radar Podcast to track the technologies and people that will shape our world in the years to come.

In this week’s episode of the Radar Podcast, O’Reilly’s Courtney Nash chats with Alois Reitbauer, chief evangelist at Ruxit, about how DevOps is deeply woven into the Ruxit culture. Reitbauer also talks about how the term “performance hacking” came about at Ruxit, why Chaos Monkey should be applied to systems of people, and why trust across — and between — all departments in an organization is essential.

DevOps beyond DevOps

Performance hacking is a term that emerged at Ruxit about a year ago as the company prepared for the beta launch, Reitbauer said, when they realized as the company scaled up, they needed to bring everyone from all their teams together. “The idea of performance hacking, then,” he noted, “was really, how can we scale up this collaboration between the DevOps teams, the product development teams, and our go-to-market growth hacking teams while we grow as an organization.” The ultimate aim was to figure out how to continue to act like a three-person, one-room startup as the company scaled to a couple hundred people.

One of the approaches embraced at Ruxit is to apply some of the DevOps best practices to their growth hacking and product development strategies. As an example, Reitbauer offered up the idea of Chaos Monkey, as applied not to AWS instances, but to the organization as a whole. The way it works, he explained, is to send a member of a team — any team — away on short notice (or no notice) and see what breaks. Reitbauer said that they’d actually done this and outlined what they learned from the exercise:

We have done it, and it also came up as part of our regular organizational practices. Like, when we had our first very strong conference season — we sent people to different shows all over the place; we picked people from the team who had to go somewhere. And even if people knew they were going to be out of the office in a couple of weeks, they still started to behave as if they would be around all the time, that they wouldn’t be leaving the office. Then, suddenly they were on a plane. They had no time to do their everyday work, and suddenly they realized where they really needed help. So, you don’t even have to make it a total surprise. You just have to plan how to get people out of their regular working behavior to do something else. Then we were able to figure out, ‘We really need somebody else to be able to jump in here or to help somebody out over there.’ It might be a regular holiday or just not daily routine.

Read more…

Comment: 1

The Puppet design philosophy

Explore the declarative, idempotent, and stateless Puppet DSL.

It can be very easy to get started with Puppet, but scaling it effectively can be a challenge. In this early release excerpt from Puppet Best Practices, author Chris Barbour discusses the philosophy behind using Puppet effectively.

Before we begin to explore practical best practices with Puppet, it’s valuable to understand the reasoning behind these recommendations.

Puppet can be somewhat alien to technologists who have a background in automation scripting. Where most of our scripts are procedural, Puppet is declarative. While a declarative language has many major advantages for configuration management, it does impose some interesting restrictions on the approaches we use to solve common problems.

Although Puppet’s design philosophy may not be the most exciting topic to begin this book, it drives many of the practices in the coming chapters. Understanding that philosophy will help contextualize many of the recommendations covered.

Read more…
Comments: 7
Four short links: 2 April 2015

Four short links: 2 April 2015

250 Whys, Amazon Dash, Streaming Data, and Lightning Networks

  1. What I Learned from 250 WhysLet’s Plan for a Future Where We’re All As Stupid as We Are Today.
  2. Thoughts on Amazon Dash (Matt Webb) — In a way, we’re really seeing the future of marketing here. We’ve separated awareness (advertising) and distribution (stores) for so long, but it’s no longer the way. When you get a Buy Now button in a Tweet, you’re seeing ads and distribution merging, and the Button is the physical instantiation of this same trend. […] in the future every product will carry a buy button.
  3. A Collection of Links for Streaming Algorithms and Data Structures — is this not the most self-evident title ever?
  4. Lightning Networks (Rusty Russell) — I finally took a second swing at understanding the Lightning Network paper. The promise of this work is exceptional: instant, reliable transactions across the bitcoin network. But the implementation is complex, and the draft paper reads like a grab bag of ideas; but it truly rewards close reading! It doesn’t involve novel crypto, nor fancy bitcoin scripting tricks. There are several techniques that are used in the paper, so I plan to concentrate on one per post and wrap up at the end. Already posted part II.
Four short links: 31 March 2015

Four short links: 31 March 2015

Boring Technology, Psychology Memes, Engineering Ladder, and Flatpack Refugee Shelters

  1. Choose Boring Technology (Dan McKinley) — Adding technology to your company comes with a cost. As an abstract statement this is obvious: if we’re already using Ruby, adding Python to the mix doesn’t feel sensible because the resulting complexity would outweigh Python’s marginal utility. But somehow when we’re talking about Python and Scala or MySQL and Redis, people lose their minds, discard all constraints, and start raving about using the best tool for the job.
  2. Dunning-Kruger and Other Memes — a reality check on the popsci conception of some psych research.
  3. Sharing our Engineering LadderIn addition to the ladder causing problems inside of my team, we were having a hard time evaluating candidates during interviews and determining what level to hire them into. Particularly at the more senior levels, it wasn’t clear what the criteria for success really looked like. So, together with my tech leads and engineering managers, we rewrote the ladder to be more specific. It has been very helpful both for the process of reviews and promotion committees as well as for the process of hiring.
  4. Ikea’s flat-pack refugee shelter is entering production (The Verge) — The UNHCR has agreed to buy 10,000 of the shelters, and will begin providing them to refugee families this summer. […] Measuring about 188 square feet, each shelter accommodates five people and includes a rooftop solar panel that powers a built-in lamp and USB outlet. The structure ships just like any other piece of Ikea furniture, with insulated, lightweight polymer panels, pipes, and wires packed into a cardboard box. According to Ikea, it only takes about four hours to assemble.

How the DevOps revolution informs software architecture

The O'Reilly Radar Podcast: Neal Ford on the changing role of software architects and the rise of microservices.


In this episode of the Radar Podcast, O’Reilly’s Mac Slocum sits down with Neal Ford, a software architect and meme wrangler at ThoughtWorks, to talk about the changing role of software architects. They met up at our recent Software Architecture Conference in Boston — if you missed the event, you can sign up to be notified when the Complete Video Compilation of all sessions and talks is available.

Slocum started the conversation with the basics: what, exactly, does a software architect do. Ford noted that there’s not a straightforward answer, but that the role really is a “pastiche” of development, soft skills and negotiation, and solving business domain problems. He acknowledged that the role historically has been negatively perceived as a non-coding, post-useful, ivory tower deep thinker, but noted that has been changing over the past five to 10 years as the role has evolved into real-world problem solving, as opposed to operating in abstractions:

“One of the problems in software, I think, is that you build everything on towers of abstractions, and so it’s very easy to get to the point where all you’re doing is playing with abstractions, and you don’t reify that back to the real world, and I think that’s the danger of this kind of ivory-tower architect. When you start looking at things like continuous delivery and continuous deployment, you have to take those operational concerns into account, and I think that is making the role of architect a lot more relevant now, because they are becoming much more involved in the entire software development ecosystem, not just the front edge of it.”

Read more…

Comment: 1
Four short links: 19 March 2015

Four short links: 19 March 2015

Changing Behaviour, Building Filters, Public Access, and Working Capital

  1. Using Monitoring Dashboards to Change Behaviour[After years of neglect] One day we wrote some brittle Ruby scripts that polled various services. They collated the metrics into a simple database and we automated some email reports and built a dashboard showing key service metrics. We pinpointed issues that we wanted to show people. Things like the login times, how long it would take to search for certain keywords in the app, and how many users were actually using the service, along with costs and other interesting facts. We sent out the link to the dashboard at 9am on Monday morning, before the weekly management call. Within 2 weeks most problems were addressed. It is very difficult to combat data, especially when it is laid out in an easy to understand way.
  2. Quiet Mitsubishi Cars — noise-cancelling on phone calls by using machine learning to build the filters.
  3. NSF Requiring Public AccessNSF will require that articles in peer-reviewed scholarly journals and papers in juried conference proceedings or transactions be deposited in a public access compliant repository and be available for download, reading, and analysis within one year of publication.
  4. Filtered for Capital (Matt Webb) — It’s important to get a credit line [for hardware startups] because growing organically isn’t possible — even if half your sell-in price is margin, you can only afford to grow your batch size at 50% per cycle… and whether it’s credit or re-investing the margin, all that growth incurs risk, because the items aren’t pre-sold. There are double binds all over the place here.

Worship maintainers

The future is maintenance: build for the inevitable.


Technology has had a cult of newness for centuries. We hail innovators, cheer change, and fend off critics who might think new and change are coming too fast. Unfortunately, while that drives the cycle of creation, it also creates biases that damage what we create, reducing the benefits and increasing the costs.

Formerly new things rapidly become ordinary “plumbing,” while maintenance becomes a cost center, something to complain about. “Green fields” and startups look ever more attractive because they offer opportunities to start fresh, with minimal connections to past technology decisions.

The problem, though, is that most of these new things — the ones that succeed enough to stay around — have a long maintenance cycle ahead of them. As Axel Rauschmayer put it:

“People who maintain stuff are the unsung heroes of software development.”

In a different context, Steve Hendricks of Historic Doors pointed out that:

“Low maintenance is the holy grail of our culture. We’ve gone so far that we’re willing to throw things away rather than fix them.”

That gets especially expensive. Heaping praise on the creators of new things while trying to minimize the costs of the maintainers is a recipe for disaster over the long term.
Read more…

Comment: 1
Four short links: 17 March 2015

Four short links: 17 March 2015

Open Source Personal Assistant, Flintstoning Robots, Year of Personal Assistants, and Infrastructure Curiosity

  1. Sirius — UMich open source “intelligent Personal Assistant” (aka Siri, Cortana, Google Now, etc.). Text recognition, image recognition, query processing components. They hope it’ll be a focal point for research in the area, the way that open source operating systems have focused university research.
  2. MIT DragonBot Evolving to Teach Kids (IEEE Spectrum) — they’re moving from “Wizard of Oz” (humans-behind-the-scenes) control to autonomous operation. Lovely example of Flintstoning in a robotics context.
  3. Personal Assistants Coming (Robohub) — 2015 is the year physical products will be coming to market and available for experimentation and testing. Pepper ships in the summer in Japan, JIBO ships preorders in Q3, as does Cubic in the fall and EmoSpark in the summer. […]The key to the outcome of this race is whether a general purpose AI will be able to steer people through their digital world, or whether users would rather navigate to applications that are specialists (such as American Airlines or Dominos Pizza).
  4. Incuriosity Killed the Infrastructurebeing actively curious about “fishy” things will lead to a more stable and happy infrastructure.
Four short links: 27 February 2015

Four short links: 27 February 2015

No Estimates, Brand Advertising, Artificial Intelligence, and GPG BeGone

  1. #NoEstimatesAllspaw also points out that the yearning to break the bonds of estimation is nothing new — he’s fond of quoting a passage from The Unwritten Laws of Engineering, a 1944 manual which says that engineers “habitually try to dodge the irksome responsibility for making commitments.” All of Allspaw’s segment is genius.
  2. Old Fashioned Snapchatget a few drinks in any brand advertiser and they’ll admit that the number one reason they know that brand advertising works is that, if they stop, sales inevitably drop.
  3. Q&A With Bruce Sterling on Artificial Intelligence — in which Sterling sounds intelligent, and the questioner sounds Artificial.
  4. GPG and Me (Moxie Marlinspike) — Even though GPG has been around for almost 20 years, there are only ~50,000 keys in the “strong set,” and less than 4 million keys have ever been published to the SKS keyserver pool ever. By today’s standards, that’s a shockingly small user base for a month of activity, much less 20 years. This was a great talk at Webstock this year.
Four short links: 26 February 2015

Four short links: 26 February 2015

Autocompletion, Colliding Trends, Microservices, and Writing Useful Code

  1. awesomplete — MIT-licensed ultra lightweight, usable, beautiful autocomplete with zero dependencies in Javascript.
  2. How to Seize the Opportunities when Megatrends Collide — excuse the cheesy title, the chart from PwC showing pairwise combination of trends, is interesting.
  3. Adopting Microservices at Netflix: Lessons for Architectural Designyou want to think of servers like cattle, not pets. If you have a machine in production that performs a specialized function, and you know it by name, and everyone gets sad when it goes down, it’s a pet. Instead you should think of your servers like a herd of cows. What you care about is how many gallons of milk you get. If one day you notice you’re getting less milk than usual, you find out which cows aren’t producing well and replace them. People for Ethical Treatment of Iron, your time has come!
  4. Your Job is Not to Write Code (Laura Klein) — I know what you’re thinking. This will all take so long! I’ll be so much less effective! This isn’t true. You’ll be far more effective because you will actually be doing your job. Amen to it all.