"CI" entries

DevOps for Finance

How DevOps will help you surpass the common challenges of financial services software development.

Download a free copy of DevOps for Finance, an O’Reilly report by Jim Bird for the financial services software insider who’s heard about DevOps, but is unsure whether it represents solution or suicide.

DevOps, until recently, has been a story about unicorns. Innovative, engineering-driven online tech companies like Flickr, Etsy, Twitter, Facebook, and Google. Netflix and its Chaos Monkey. Amazon deploying thousands of changes per day.

DevOps was originally about WebOps at Internet companies working in the Cloud, and a handful of Lean Startups in Silicon Valley. It started at these companies because they had to move quickly, so they found new, simple, and collaborative ways of working that allowed them to innovate much faster and to scale much more effectively than organizations had done before.

But as the velocity of change in business continues to increase, enterprises — sometimes referred to as “horses,” in contrast to the unicorns referenced above — must also move to deliver content and features to customers just as quickly. These large organizations have started to adopt (and, along the way, adapt) DevOps ideas, practices, and tools.

This short book assumes that you have heard about DevOps and want to understand how DevOps practices like Continuous Delivery and Infrastructure as Code can be used to solve problems in financial systems at a trading firm, or a big bank or stock exchange. We’ll look at the following key ideas in DevOps, and how they fit into the world of financial systems:

  • Breaking down the “wall of confusion” between development and operations, and extending Agile practices and values from development to operations
  • Using automated configuration management tools like Chef, Puppet, CFEngine, or Ansible to programmatically provision and configure systems (Infrastructure as Code)
  • Building Continuous Integration and Continuous Delivery (CI/CD) pipelines to automatically test and push out changes, and wiring security and compliance into these pipelines
  • Using containerization and virtualization technologies like Docker and Vagrant, together with Infrastructure as Code, to create IaaS, PaaS, and SaaS clouds
  • Running experiments, creating fast feedback loops, and learning from failure

To follow this book you need to understand a little about these ideas and practices. There is a lot of good stuff about DevOps out there, amid the hype. Read more…


A software engineer’s role traversal

Software engineer and author Jason Myers on changing roles in a changing market.


We often hear about how the tech job market is booming and has space for newcomers, but what does that mean for the developers already in the market? In December of 2014, Fortune.com predicted that 2015 would be an excellent year for developers to change jobs. Citing Dice.com, they note that jobs are popping up all over the country. In fact, Dice’s survey also reports 40% of hiring managers seeing voluntary departures, a higher number than was seen just six months earlier.

These are all large, general numbers. What does a job change, and the changing market, look like for individual developers? To get a better sense, I spoke with Jason Myers, who is working on our upcoming Essential SQLAlchemy, 2e title. Jason recently went from working for the email marketing service Emma, Inc., to working for networking giant Cisco. Here, he talks about how a change like that feels, and how the market looks to him.

Read more…


How CI removes the pain

The tradeoffs of (accidentally) discarding continuous integration.

Engineering Practices for Continuous Delivery

I’ve given a continuous delivery workshop a few times with ThoughtWorks Chief Scientist Martin Fowler, who tells an interesting story about continuous integration, from the first software project he ever saw. When Martin was a teenager, his father had a friend who was running a software project, and he gave Martin the nickel (or five pence) tour — a bunch of men, predominately on mainframe terminals, working in an old warehouse. Martin remarked that the thing that struck him the most was when the guide told him that all the developers were “currently integrating all their code.” They had finished coding six months prior, yet despite that they weren’t sure when they were going to be done “integrating.”

That revelation surprised Martin: in his mind, software development was a discreet, scientific, deterministic process, not at all represented by the vague comments of these developers. But the practice of as-late-as-possible integration was common back in an earlier era of software development. If you look at software engineering texts of the ’60s and ’70s, every project included an integration phase. This isn’t how we think of integration today, which happens at the granularity of services and applications. Rather, a common practice was to have developers code in isolation for weeks and months at a time, then integrate all their code together into a cohesive whole. And that phase was, not surprisingly, a painful part of most projects. Yet, that type of 60s and 70s workflow is still codified in some version control tools today, even though we have now determined that late integration is the opposite of how we should approach this problem.

Read more…

Comments: 4

What is DevOps (yet again)?

Empathy, communication, and collaboration across organizational boundaries.

Cropped image "Kilobot robot swarm" by asuscreative - Own work. Licensed under CC BY-SA 4.0 via Wikimedia Commons.http://commons.wikimedia.org/wiki/File:Kilobot_robot_swarm.JPG#mediaviewer/File:Kilobot_robot_swarm.JPG

I might try to define DevOps as the movement that doesn’t want to be defined. Or as the movement that wants to evade the inevitable cargo-culting that goes with most technical movements. Or the non-movement that’s resisting becoming a movement. I’ve written enough about “what is DevOps” that I should probably be given an honorary doctorate in DevOps Studies.

Baron Schwartz (among others) thinks it’s high time to have a definition, and that only a definition will save DevOps from an identity crisis. Without a definition, it’s subject to the whims of individual interest groups, and ultimately might become a movement that’s defined by nothing more than the desire to “not be like them.” Dave Zwieback (among others) says that the lack of a definition is more of a blessing than a curse, because it “continues to be an open conversation about making our organizations better.” Both have good points. Is it possible to frame DevOps in a way that preserves the openness of the conversation, while giving it some definition? I think so.

DevOps started as an attempt to think long and hard about the realities of running a modern web site, a problem that has only gotten more difficult over the years. How do we build and maintain critical sites that are increasingly complex, have stringent requirements for performance and uptime, and support thousands or millions of users? How do we avoid the “throw it over the wall” mentality, in which an operations team gets the fallout of the development teams’ bugs? How do we involve developers in maintenance without compromising their ability to release new software?

Read more…

Comments: 2

The DevOps identity crisis

Why DevOps needs a manifesto after all, but may never get one.

Image: CC BY-SA 2.0 Libby Levi for opensource.com

DevOps is everywhere! The growth and mindshare of the movement is remarkable. But if you care deeply about DevOps, you might agree with me when I say that although its moment has “arrived,” DevOps is in serious trouble. The movement is fragmented and weakly defined, and is being usurped by those who care more about short-term opportunities than the long-term viability of DevOps.

They are the ninety-nine percent, and nobody cares

How bad could it be? Travel back in time. It is mid-November 2011, and Occupy Wall Street is occupying the headlines. One of the major reasons is that the protestors are targeting shipping ports on the West Coast, causing shutdowns and even violence. As things are getting out of hand, parts of the movement start condemning these actions as counter-productive, hurting the 99% instead of the intended 1%. Spokespeople for the movement are quoted in the media as saying the instigators “don’t represent the movement.”

Why did the Occupy movement become a footnote in history so fast? There were several reasons: there was no cohesive agreement on its identity, values, goals, and mission; in an effort to be unlike “them,” the OWS proponents avoided anything that looked like centralized leadership; and it seemed to be entirely negative, advocating nothing to replace what it wanted to remove.

I believe a similar thing is happening to DevOps right now, for many of the same reasons. Let’s talk about some of these problems.

Read more…

Comments: 16

Introducing “A Field Guide to the Distributed Development Stack”

Tools to develop massively distributed applications.

Editor’s Note: At the Velocity Conference in Barcelona we launched “A Field Guide to the Distributed Development Stack.” Early response has been encouraging, with reactions ranging from “If I only had this two years ago” to “I want to give a copy of this to everyone on my team.” Below, Andrew Odewahn explains how the Guide came to be and where it goes from here.

As we developed Atlas, O’Reilly’s next-generation publishing tool, it seemed like every day we were finding interesting new tools in the DevOps space, so I started a “Sticky” for the most interesting-looking tools to explore.


At first, this worked fine. I was content to simply keep a list, where my only ordering criteria was “Huh, that looks cool. Someday when I have time, I’ll take a look at that,” in the same way you might buy an exercise DVD and then only occasionally pull it out and think “Huh, someday I’ll get to that.” But, as anyone who has watched DevOps for any length of time can tell you, it’s a space bursting with interesting and exciting new tools, so my list and guilt quickly got out of hand.

Read more…