To all of you that care so much about building & running the infrastructure that we depend on every day… thank you. You are exceptional people, and you are doing work that matters.

Here’s a video of Tim O’Reilly talking about how he came to learn how awesome SysAdmins are.

]]> http://radar.oreilly.com/2011/07/happy-sysadmin-appreciation-da.html/feed 3 http://radar.oreilly.com/2011/04/facebook-open-compute-ops.html http://radar.oreilly.com/2011/04/facebook-open-compute-ops.html#comments Thu, 07 Apr 2011 21:00:00 +0000 Jesse Robbins http://blogs.oreilly.com/radar/2011/04/facebook-open-compute-ops.html Today, Jonathan Heiliger, VP of Operations at Facebook, and his team announced the Open Compute Project, releasing their data center hardware stack as open source. This is a revolutionary project, and I believe it’s one of the most important in infrastructure history. Let me explain why.

The way we operate systems and datacenters at web scale is fundamentally different than the world most server vendors seem to design their products to run in.

Web-scale systems focus on the entire system as a whole. In our world, individual servers are not special, and treating them as special can be dangerous. We expect servers to fail and we increasingly rely on the software we write to manage those failures. In many cases, the most valuable thing we can do when hardware fails is to simply provision a new one as quickly as possible. That means having enough capacity to do that, a way of programmatically managing the infrastructure, and an easy way to replace the failed components.

The server vendors have been slow to make this transition because they have been focused on individual servers, rather than systems as a whole. What we want to buy is racks of machines, with power and networking preconfigured, which we can wheel in, bolt down, and plug in. For the most part we don’t care about logos, faceplates, and paint jobs. We won’t use complex integrated proprietary management interfaces, and we haven’t cared about video cards in a long time … although it is still very hard to buy a server without them.

This gap is what led Google to build their own machines optimized for their own applications in their own datacenters. When Google did this, they gained a significant competitive advantage. Nobody else could deploy as much compute power as quickly and efficiently. To complete with Google’s developers you also must compete with their operations and data center teams. As Tim O’Reilly said: “Operations is the new secret sauce.”

When Jonathan and his team set out to build Facebook’s new datacenter in Oregon, they knew they would have to do something similar to achieve the needed efficiency. Jonathan says that the Prineville, Ore. data center uses 38% less energy to do the same work as Facebook’s existing facilities, while costing 24% less.

Facebook then took the revolutionary step of releasing the designs for most of the hardware in the datacenter under the Creative Commons license. They released everything from the power supply and battery backup systems to the rack hardware, motherboards, chassis, battery cabinets, and even their electrical and mechanical construction specifications.

This is a gigantic step for open source hardware, for the evolution of the web and cloud computing, and for infrastructure and operations in general. This is the beginning of a shift that began with open source software, from vendors and consumers to a participatory and collaborative model. Jonathan explains:

“The ultimate goal of the Open Compute Project, however, is to spark a collaborative dialogue. We’re already talking with our peers about how we can work together on Open Compute Project technology. We want to recruit others to be part of this collaboration — and we invite you to join us in this mission to collectively develop the most efficient computing infrastructure possible.”

At the announcement this morning, Graham Weston of Rackspace announced that they would be participating in Open Compute, which is an ideal compliment to the OpenStack cloud computing projects. Representatives from Dell and HP spoke at the announcement and also said that they would participate in this new project. The conversation has already begun.

]]> http://radar.oreilly.com/2010/06/creating-cultural-change.html/feed 2 http://radar.oreilly.com/2010/06/velocity-web-performance-operations-webops-devops.html http://radar.oreilly.com/2010/06/velocity-web-performance-operations-webops-devops.html#comments Thu, 03 Jun 2010 00:30:00 +0000 Jesse Robbins http://blogs.oreilly.com/radar/2010/06/velocity-web-performance-operations-webops-devops.html Velocity 2010 is happening on June 22-24 (right around the corner!). This year we've added third track, Velocity Culture, dedicated to exploring what we've learned about how great teams and organizations work together to succeed at scale.

Web Operations, or WebOps, is what many of us have been calling these ideas for years. Recently the term "DevOps" has become a kind of rallying cry that is resonating with many, along with variations on Agile Operations. ]]> Velocity 2010 is happening on June 22-24 (right around the corner!). This year we’ve added third track, Velocity Culture, dedicated to exploring what we’ve learned about how great teams and organizations work together to succeed at scale.

Web Operations, or WebOps, is what many of us have been calling these ideas for years. Recently the term “DevOps” has become a kind of rallying cry that is resonating with many, along with variations on Agile Operations. No matter what you call it, our experiences over the past decade taught us that Culture matters more than any tool or technology in building, adapting, and scaling the web.

Here is a small sample of the upcoming Velocity Culture sessions:

Ops Meta-Metrics: The Currency You Use to Pay For Change

Presenter: John Allspaw (Etsy.com)

Change to production environments can cause a good deal of stress and strain amongst development and operations teams. More and more organizations are seeing benefits from deploying small code changes more frequently, for stability and productivity reasons. But how can you figure out how much change is appropriate for your application or your culture?

A Day in the Life of Facebook Operations

Presenter:
Tom Cook (Facebook)

Facebook’s Technical Operations team has to balance this need for constant availability with a fast-moving and experimental engineering culture. We release code every day. Additionally, we are supporting exponential user growth while still managing an exceptionally high radio of users per employee within engineering and operations.

This talk will go into how Facebook is “run” day-to-day with particular focus on actual tools in use (configuration management systems, monitoring, automation, etc), how we detect anomalies and respond to them, and the processes we use internally for rapidly pushing out changes while still keeping a handle on site stability.

Change Management: A Scientific Classification

Presenter: Andrew Shafer (Cloudscaling)

Change management is the combination of process and tools by which changes are made to production systems. Approaches range from cowboy style, making changes to the live site, to complex rituals with secret incantations, coming full circle to continuous deployment. This presentation will highlight milestone practices along this spectrum, establishing a matrix for evaluating deployment process.

There is a tremendous amount happing in our space in the coming weeks in addition to the conference itself. First, the “Web Operations” book that John Allspaw and I edited goes to print on June 15th. We’re really excited about how it came together. Then, immediately after Velocity is DevOpsDays, which is a great community event that continues the conversation after Velocity (and is free). Hope to see you all there!

Related:

• How Facebook satisfied a need for speed
• Online video primer: Buffering, streaming, and infrastructure

Over the past three years I have been working to bridge gaps between the tech community & traditional emergency management organizations.  I’ve focused on helping technologists adapt technologies to support humanitarian missions, often in response to a disaster.  
After Hurricane Katrina, Mikel Maron and I discovered a pattern for successful innovation during and after disasters.  Understanding this pattern is crucial to “Serving Those that Serve Others”.
Pattern for DisasterTech Innovation 1. Disaster2. Ad-Hoc Adaptation 3. Championship 4. Iterative Improvement
There is an unprecedented amount of interest and attention in finding ways to help in Haiti & around the world.  The CrisisCamp & CrisisCommon projects are coordinating events and helping match organizations with needs to volunteers with skills.  I encourage you to participate, and volunteer your time, knowledge, and resources.  
Serve those that serve others.  You can make a difference now.

Upcoming Crisis Camps

Steve Souders & I have set a new theme this year, “Fast by Default”.  We want the broader Velocity community & to adopt it as a shared mission & mantra. The reason for this is simple…

Fast isn’t a Feature. Fast is a Requirement.

At Velocity earlier this year Marissa Meyer explained why performance mattered so much to Google. Then Eric Schurman (Bing & Velocity Program Committee member) and Jake Brutlag (Google Search) made history with a co-presentation on just how crucial performance is to revenue .

Phil Dixon of Shopzilla explained that a 5 second performance improvement increased their revenue by 7-12 percent while reducing hardware spend by 50%!!!

Fast means Client, Server, Infrastructure, Operations, & Organizations

Getting to Fast isn’t just about any one part of the system. Browser & Client performance is crucial, and requires an equally fast server & infrastructure to support it. When load increases, infrastructure must scale quickly or performance suffers. The operational tools and processes for managing software & infrastructure must support rapid changes in a dynamic environment, and be backed by an organization & culture that embraces it.

We’re Looking for Speakers – Submit your Proposals Now!

Do you have ideas and experience for improving Web Performance & Operations and making things “Fast by Default”? We want you as a speaker at Velocity 2010.

Submit your Proposals Now! Entires are due no later than January 11th, 2010 at 11:59 PM Pacific.

One more thing…

Quite a few people have asked us to have Velocity conferences more frequently & beyond the SF Bay Area, and so we’re going to try something new. On December 8 we’ll be running our first ever Velocity Online Conference.

Past Velocity Conference participants get a 50% discount & get a 25% discount off Velocity 2010.

See the full schedule after the jump…

Their data showed that slow sites get fewer search queries per user, less revenue per visitor, fewer clicks, fewer searches, and lower search engine rankings. They found that in some cases even after site performance was improved users continued to interact as if it was slow. Bad experiences have a lasting influence on customer behavior.

What about smaller websites that aren’t yet at this scale?

Alistair Croll and Sean Power, the authors of the new book Complete Web Monitoring, have continued this research for sites at smaller scale.

They used a Strangeloop Networks web acceleration appliance to optimize half the sessions to a smaller production website, tagging optimized and unoptimized visitors so they could be analyzed in Google Analytics. The Strangeloop device applies many of Steve Souders’ performance rules to an existing site automatically (a kind of “Steve-in-a-Box” ;-).

The results of their analysis show how significant a reduction in page latency can be. In addition to reducing bounce rates, and increasing pages per visit & time on site, they found a 16.07% increase in conversion rates and a 5.50% increase in average order value.

Check out the full post on the Watching Websites blog.

Good luck today John & team… I know it’s going to be a long day!

Update: Apparently Facebook & Livejournal have had similar attacks today. Rich Miller from Data Center Knowledge reminds us that this is just the latest in a series of major attacks.

Here is his keynote in its entirety:

Note: Other videos from Velocity are being posted to VelocityConference.blip.tv

The deadline for talks is May 11th, so submit your talks now!

As with all Ignites each speaker will only get 20 slides that each auto-advance every 15 seconds for a total of five minutes. We’ll be looking for fun geek topics like hacks, how-to’s, and insights. (Talks don’t have to be Velocity-related!) If you’re not sure what an Ignite talk looks like check out the Ignite Show.

You can RSVP for the event on Upcoming or Facebook.