From the network interface to the database

All systems are distributed systems, and we’re starting to see how they fit into Velocity's themes.


From the beginning, the Velocity Conference has focused on web performance and operations — specifically, web operations. This focus has been fairly narrow: browser performance dominated the discussion of “web performance,” and interactions between developers and IT staff dominated operations.

These limits weren’t bad. Perceived performance really is dominated by the browser — how fast you can get resources (HTML, images, CSS files, JavaScript libraries) over the network to the browser, and how fast the browser can execute those resources. How long before a user stops waiting for your page to load and clicks away? How do you make a page useable as quickly as possible, even before all the resources have loaded? Those discussions were groundbreaking and surprising: users are incredibly sensitive to page speed.

That’s not to say that Velocity hasn’t looked at the rest of the application stack; there’s been an occasional glance in the direction of the database and an even more occasional glance at the middleware. But the database and middleware have, at least historically, played a bit part. And while the focus of Velocity has been front-end tuning, speakers like Baron Schwartz haven’t let us ignore the database entirely.

The web operations side of Velocity has been more diverse: integrating the work of developers and ops staff, moving from waterfall practices to “agile” development, developing a culture of continuous deployment — these have been major milestones in the Velocity story. We’re proud that a healthy “devops” movement grew out of Velocity. But here, too, there’s certainly a bigger story to tell.

In the operations world, it’s never been possible to abstract a system from what’s happening at the lowest levels. In the past few years, we’ve learned that all applications are distributed. So, there have been sessions on resilient systems, accepting failure, and blameless postmortems: that’s a start. There’s a lot of system between the web server and the browser. For that matter, there’s a lot of system between the web server and the point where the bits leave the building. For that matter squared, what building? A building you own, or a building Amazon owns in a city you can’t name?

While Velocity has been a pioneer in recognizing the distributed nature of modern computing as well as the tools and the cultural changes needed to deal effectively with a distributed world, we’ve only taken occasional glances at the infrastructure that lies behind the server. We’ve only taken occasional looks at data centers themselves. And I don’t think we’ve ever discussed routing issues or routers, though a router failure can sink your application just as badly as a dead server.

So, in our ongoing exploration of web performance and operations, we’re going to broaden the scope. I don’t think we’ll be doing less of anything than we are today, but we do have to take a step back and look at the bigger picture. If everything is distributed, it makes no sense to look at part of a distributed system and skip the rest. In particular:

  • What are the performance and operational implications of our low-level network infrastructure? What happens when you hand off your packets to “the network”? There are many important questions here. How can you use new technologies like software-defined networks and network function virtualization to make your infrastructure more reliable? What roles do CDNs and other intermediaries play in delivering data to our users? How does the advent of a “slow lane” for those of us who aren’t rich enough to negotiate with Comcast, Verizon, and AT&T affect our applications? We don’t have to like it, but we’re naive if we don’t think those issues will affect us.
  • What are the performance and operational implications of middleware and databases? While the back end of the application doesn’t have the same millisecond-by-millisecond effect on performance, it has a huge effect on scalability. And an application that hasn’t scaled well is very slow. We’re not talking about the extra second of latency that makes some users frustrated and drives them away: we’re talking about being dead in the water during the Christmas rush. An application that’s slow because it can’t handle peak loads is slow in an entirely different way from an application that downloads 5 MB of JavaScript libraries and images before it’s useable, but the users don’t care either way. It’s slow, and they’re going elsewhere.

All systems are distributed systems. And in that sense, there’s nothing really new here. Rather than focusing narrowly on a few key components of our distributed systems, we’re extending the scope to include the whole thing, even the parts we don’t know about or see. Furthermore, as our distributed systems evolve, we see how they fit into Velocity’s historical themes. Configuration is code, but that’s a nasty argument to make when the “code” you’re talking about is a mess of Cisco IOS configuration files. Software-defined networks (and their descendants) turn network configuration into software. And in a virtualized, cloud-oriented world, automated database scaling is also a matter of software.

The historical mission of Velocity has been to unite web developers and operations, to get both sides to realize they’re on the same team and talking the same language. Over the years, the number of people we’ve invited to the conversation has grown: a few years ago, we started to address mobile developers. Now, the conversation is becoming even wider. We trust we won’t lose focus. But if there’s one thing we’ve learned over the years, silos are nobody’s friend, and it doesn’t matter who’s living in the silo, whether it’s a DBA or a router administrator.

Everything is distributed. And when everything is distributed, everyone has a stake in the conversation. We’re looking forward to burning down a few more silos, and inviting even more people into the Velocity tent. It’s a big tent indeed.

Photo by Ian Barbour, used under a Creative Commons license.


tags: ,

Get the O’Reilly Systems Engineering and Operations Newsletter

Get weekly insight from industry insiders—plus exclusive content, offers, and more on the topics of systems engineering and operations.