- Docker Secure Deployment Guidelines — deployment checklist for securely deploying Docker.
- The Devops Identity Crisis (Baron Schwartz) — I saw one framework-retailing bozo saying that devops was the art of ensuring there were no flaws in software. I didn’t know whether to cry or keep firing until the gun clicked.
- Apache Giraph — an iterative graph processing system built for high scalability. For example, it is currently used at Facebook to analyze the social graph formed by users and their connections.
- Apache Flink — a data processing system and an alternative to Hadoop’s MapReduce component. It comes with its own runtime, rather than building on top of MapReduce. As such, it can work completely independently of the Hadoop ecosystem. However, Flink can also access Hadoop’s distributed file system (HDFS) to read and write data, and Hadoop’s next-generation resource manager (YARN) to provision cluster resources. Since most Flink users are using Hadoop HDFS to store their data, we ship already the required libraries to access HDFS.
"containers" entries
How to create a Swarm cluster with Docker
Using Docker Machine to create a Swarm cluster across cloud providers.
Editor’s note: this is an Early Release excerpt from Chapter 7 of Docker Cookbook by Sébastien Goasguen. The recipes in this book will help developers go from zero knowledge to distributed applications packaged and deployed within a couple of chapters. One of the key value propositions of Docker is app portability. The following will show you how to use Docker Machine to create a Swarm cluster across cloud providers.
Problem
You understand how to create a Swarm cluster manually (see Recipe 7.3), but you would like to create one with nodes in multiple public Cloud Providers and keep the UX experience of the local Docker CLI.
Solution
Use Docker Machine to start Docker hosts in several Cloud providers and bootstrap them automatically to create a swarm cluster.
The shape of software architecture
Five things we learned from the O’Reilly Software Architecture Conference 2015.
Last week, I had the opportunity to see the first Software Architecture Conference spring to life after a winter of preparation. Software architects, with or without the official title, swarmed the halls learning from speakers and attendees alike. I count myself among the people who were learning. Many notions about this profession and skill set have become clearer to me and I’m already planning to keep the content coming. I’m also in the early stages of developing out the next Software Architecture Conference (spring 2016).
Within this piece you’ll find my takeaways and lessons learned from the event. I expect these initial impressions to both shape our upcoming exploration of software architecture and be shaped by continued shifts within software architecture.
Read more…
3 easy reasons why you’ll move your business to the cloud
Migrating to cloud-native application architectures leads to innovation.
Editor’s note: this is an advance excerpt from Chapter 1 of the forthcoming Migrating to Cloud-Native Application Architectures by Matt Stine. This report examines how the cloud enables innovation and the changes an enterprise must consider when adopting cloud-native application architectures.
Let’s examine the common motivations behind moving to cloud-native application architectures.
Speed
It’s become clear that speed wins in the marketplace. Businesses that are able to innovate, experiment, and deliver software-based solutions quickly are outcompeting those that follow more traditional delivery models.
In the enterprise, the time it takes to provision new application environments and deploy new versions of software is typically measured in days, weeks, or months. This lack of speed severely limits the risk that can be taken on by any one release, because the cost of making and fixing a mistake is also measured on that same timescale.
Internet companies are often cited for their practice of deploying hundreds of times per day. Why are frequent deployments important? If you can deploy hundreds of times per day, you can recover from mistakes almost instantly. If you can recover from mistakes almost instantly, you can take on more risk. If you can take on more risk, you can try wild experiments—the results might turn into your next competitive advantage.
The elasticity and self-service nature of cloud-based infrastructure naturally lends itself to this way of working. Provisioning a new application environment by making a call to a cloud service API is faster than a form-based manual process by several orders of magnitude. Deploying code to that new environment via another API call adds more speed. Adding self-service and hooks to teams’ continuous integration/build server environments adds even more speed. Eventually we can measure the answer to Lean guru Mary Poppendick’s question, “How long would it take your organization to deploy a change that involves just one single line of code?” in minutes or seconds.
Imagine what your team… what your business… could do if you were able to move that fast!
What is DevOps (yet again)?
Empathy, communication, and collaboration across organizational boundaries.
I might try to define DevOps as the movement that doesn’t want to be defined. Or as the movement that wants to evade the inevitable cargo-culting that goes with most technical movements. Or the non-movement that’s resisting becoming a movement. I’ve written enough about “what is DevOps” that I should probably be given an honorary doctorate in DevOps Studies.
Baron Schwartz (among others) thinks it’s high time to have a definition, and that only a definition will save DevOps from an identity crisis. Without a definition, it’s subject to the whims of individual interest groups, and ultimately might become a movement that’s defined by nothing more than the desire to “not be like them.” Dave Zwieback (among others) says that the lack of a definition is more of a blessing than a curse, because it “continues to be an open conversation about making our organizations better.” Both have good points. Is it possible to frame DevOps in a way that preserves the openness of the conversation, while giving it some definition? I think so.
DevOps started as an attempt to think long and hard about the realities of running a modern web site, a problem that has only gotten more difficult over the years. How do we build and maintain critical sites that are increasingly complex, have stringent requirements for performance and uptime, and support thousands or millions of users? How do we avoid the “throw it over the wall” mentality, in which an operations team gets the fallout of the development teams’ bugs? How do we involve developers in maintenance without compromising their ability to release new software?
What containers can do for you
Docker, Rocket, and big industry changes are making it a great time to seriously consider using containers.
If you read any IT news these days it’s hard to miss a headline about “the container revolution.” Docker’s year-and-a-half-old engine had a monopoly on the buzz until CoreOS launched its own project, Rocket, in December.
The technology behind containers can seem esoteric, but the advantages of bringing containers to your organization are more compelling than ever. And containers’ inherent portability opens up exciting new opportunities for how organizations host their applications.
Containerization is having its moment and there’s never been a better time to check it out for yourself.
Four short links: 15 January 2015
Secure Docker Deployment, Devops Identity, Graph Processing, and Hadoop Alternative
Introducing “A Field Guide to the Distributed Development Stack”
Tools to develop massively distributed applications.
Editor’s Note: At the Velocity Conference in Barcelona we launched “A Field Guide to the Distributed Development Stack.” Early response has been encouraging, with reactions ranging from “If I only had this two years ago” to “I want to give a copy of this to everyone on my team.” Below, Andrew Odewahn explains how the Guide came to be and where it goes from here.
As we developed Atlas, O’Reilly’s next-generation publishing tool, it seemed like every day we were finding interesting new tools in the DevOps space, so I started a “Sticky” for the most interesting-looking tools to explore.
At first, this worked fine. I was content to simply keep a list, where my only ordering criteria was “Huh, that looks cool. Someday when I have time, I’ll take a look at that,” in the same way you might buy an exercise DVD and then only occasionally pull it out and think “Huh, someday I’ll get to that.” But, as anyone who has watched DevOps for any length of time can tell you, it’s a space bursting with interesting and exciting new tools, so my list and guilt quickly got out of hand.
Working like a startup at IBM
How a small and passionate team used modern techniques to shift a business on a short timeline.
Over the past year, I assisted in creating an application that helped shift a major part of IBM to a software-as-a-service (SaaS) model. I did this with the help of a small but excellent development team that was inspired by the culture and practices of web startups. To be clear, it wasn’t easy – changing how we worked led to frequent friction and conflict – but in the end it worked, and we made a difference.
In mid-2013, the IBM Service Management business and engineering leaders decided to make a big bet on moving our software to the cloud. Traditionally we have sold “on premises” software products. These are software products that a customer buys, downloads, and installs on their own equipment, in their own data centers and facilities. Although we love the on-premises business, we realized that cloud delivery of software is also a great option, and as our customers evolved to a hybrid on-premises / cloud future, we needed to be there to help them.
Four short links: 13 June 2014
Decentralized Web, Reproducibility Talk, Javascript Microcontroller, and Docker Maturity
- Mapping the Decentralized Movement (Jon Udell) — the pendulum is about to swing back toward a more distributed Web.
- John Ioannidis: Reproducible Research, True or False? (YouTube) — his talk at Google. (via Paul Kedrosky)
- Tessel — a microcontroller that runs Javascript. For those who can’t handle C.</troll>
- Docker Misconceptions — This is not impossible and can all be done – several large companies are already using Docker in production, but it’s definitely non-trivial. This will change as the ecosystem around Docker matures (via Flynn, Docker container hosting, etc), but currently if you’re going to attempt using Docker seriously in production, you need to be pretty skilled at systems management and orchestration.
Four short links: 10 June 2014
Trusting Code, Deep Pi, Docker DevOps, and Secure Database
- Trusting Browser Code (Tim Bray) — on the fundamental weakness of the ‘net as manifest in the browser.
- Deep Learning in the Raspberry Pi (Pete Warden) — $30 now gets you a computer you can run deep learning algorithms on. Awesome.
- Announcing Docker Hub and Official Repositories — as Docker went 1.0 and people rave about how they use it, comes this. They’re thinking hard about “integrating into the build ship run loop”, which aligns well with DevOps-enabling tool use.
- Apple’s Secure Database for Users (Ian Waring) — excellent breakdown of how Apple have gone out of their way to make their cloud database product safe and robust. They may be slow to “the cloud” but they have decades of experience having users as customers instead of products.