The two campaign systems highlight the stark differences between DevOps and traditional models.
Leaving politics aside, there’s a lot that can be learned from the technical efforts of the Obama and Romney campaigns. Just about everyone agrees that the Obama campaign’s Narwhal project was a great success, and that Romney’s Orca was a failure. But why?
I have one very short answer. If you follow technology, you don’t have to read between the lines much to realize that Narwhal embodied the best of the DevOps movement: rapid iteration, minimal barriers between developers and operations staff, heavy use of “cloud” technology, and constant testing to prove that you can handle outages and heavy load. In contrast, Romney’s Orca was a traditional corporate IT project gone bad. The plans were developed by non-technical people (it’s not clear what technical expertise Romney had on his team), and implemented by consultants. There were token testing efforts, but as far as I could tell, no serious attempts to break or stress the system so the team understood how to keep it running when the going got tough.
It’s particularly important to look at two factors: the way testing was done, which I’ve already mentioned, and the use of cloud computing. While Orca was “tested,” there is a big difference between passing automated test suites and the sort of game day exercise that the Narwhal team performed several times. In a game day, you’re actively trying to break the system in real time: in this case, a fully deployed copy of the actual system. You unplug the routers; you shut down the clusters; you bombard the system with traffic at inconceivable volumes. And the people responsible for keeping the system up are on the team that built it, and the team that ran it in real life. If you read the Atlantic account of Narwhal’s game day, you’ll see that it involved everyone, from the most senior developers on down, sweating it out to keep the system running in the face of disaster. They even simulated what would happen if one of Amazon’s availability zones went down, as has happened several times in the past few years (and happened again a few days before the election). Game day gave the Obama team a detailed plan for just about every conceivable outage, and the confidence that, if something inconceivable happened, they could handle it.