"Velocity" entries

Beyond the stack

The tools in the Distributed Developer's Stack make development manageable in a highly distributed environment.

Cairn at Garvera, Surselva, Graubuenden, Switzerland. The shape of software development has changed radically in the last two decades. We’ve seen many changes: the Internet, the web, virtualization, and cloud computing. All of these changes point toward a fundamental new reality: all computing has become distributed computing. The age of standalone applications has disappeared, and applications that run on a single computer are almost inconceivable. Distributed is the default; and whether an application is running on Amazon Web Services (AWS), on a private cloud, or even on a desktop or a mobile phone, it depends on the behavior of other systems and services that aren’t under the developer’s control.

In the past few years, a new toolset has grown up to support the development of massively distributed applications. We call this new toolset the Distributed Developer’s Stack (DDS). It is orthogonal to the more traditional world of servers, frameworks, and operating systems; it isn’t a replacement for the aged LAMP stack, but a set of tools to make development manageable in a highly distributed environment.

The DDS is more of a meta-stack than a “stack” in the traditional sense. It’s not prescriptive; we don’t care whether you use AWS or OpenStack, whether you use Git or Mercurial. We do care that you develop for the cloud, and that you use a distributed version control system. The DDS is about the requirements for working effectively in the second decade of the 21st century. The specific tools have evolved, and will continue to evolve, and we expect you to evolve, too. Read more…

Comment

Everything is distributed

How do we manage systems that are too large to understand, too complex to control, and that fail in unpredictable ways?

Complexity

“What is surprising is not that there are so many accidents. It is that there are so few. The thing that amazes you is not that your system goes down sometimes, it’s that it is up at all.”—Richard Cook

In September 2007, Jean Bookout, 76, was driving her Toyota Camry down an unfamiliar road in Oklahoma, with her friend Barbara Schwarz seated next to her on the passenger side. Suddenly, the Camry began to accelerate on its own. Bookout tried hitting the brakes, applying the emergency brake, but the car continued to accelerate. The car eventually collided with an embankment, injuring Bookout and killing Schwarz. In a subsequent legal case, lawyers for Toyota pointed to the most common of culprits in these types of accidents: human error. “Sometimes people make mistakes while driving their cars,” one of the lawyers claimed. Bookout was older, the road was unfamiliar, these tragic things happen. Read more…

Comments: 5
Four short links: 28 January 2014

Four short links: 28 January 2014

Client-Server, Total Information Awareness, MSFT Joins OCP, and Tissue Modelling

  1. Intel On-Device Voice Recognition (Quartz) — interesting because the tension between client-side and server-side functionality is still alive and well. Features migrate from core to edge and back again as cycles, data, algorithms, and responsiveness expectations change.
  2. Meet Microsoft’s Personal Assistant (Bloomberg) — total information awareness assistant. By Seeing, Hearing, and Knowing All, in the future even elevators will be trying to read our minds. (via The Next Web)
  3. Microsoft Contributes Cloud Server Designs to Open Compute ProjectAs part of this effort, Microsoft Open Technologies Inc. is open sourcing the software code we created for the management of hardware operations, such as server diagnostics, power supply and fan control. We would like to help build an open source software community within OCP as well. (via Data Center Knowledge)
  4. Open Tissue Wiki — open source (ZLib license) generic algorithms and data structures for rapid development of interactive modeling and simulation.
Comment: 1
Four short links: 3 January 2014

Four short links: 3 January 2014

Mesh Networks, Collaborative LaTeX, Distributed Systems Book, and Reverse-Engineering Netflix Metadata

  1. Commotion — open source mesh networks.
  2. WriteLaTeX — online collaborative LaTeX editor. No, really. This exists. In 2014.
  3. Distributed Systems — free book for download, goal is to bring together the ideas behind many of the more recent distributed systems – systems such as Amazon’s Dynamo, Google’s BigTable and MapReduce, Apache’s Hadoop etc.
  4. How Netflix Reverse-Engineered Hollywood (The Atlantic) — Using large teams of people specially trained to watch movies, Netflix deconstructed Hollywood. They paid people to watch films and tag them with all kinds of metadata. This process is so sophisticated and precise that taggers receive a 36-page training document that teaches them how to rate movies on their sexually suggestive content, goriness, romance levels, and even narrative elements like plot conclusiveness.
Comment
Four short links: 19 November 2013

Four short links: 19 November 2013

Ad Triumphalism, Education Not Transformed, Bookstore Infrastructure, and Tossable Camera

  1. Why The Banner Ad is Heroic — enough to make Dave Eggers cry. Advertising triumphalism rampant.
  2. Udacity/Thrun ProfileA student taking college algebra in person was 52% more likely to pass than one taking a Udacity class, making the $150 price tag–roughly one-third the normal in-state tuition–seem like something less than a bargain. In which Udacity pivots to hiring-sponsored workforce training and the new educational revolution looks remarkably like sponsored content.
  3. Amazon is Building Substations (GigaOm) — the company even has firmware engineers whose job it is to rewrite the archaic code that normally runs on the switchgear designed to control the flow of power to electricity infrastructure. Pretty sure that wasn’t a line item in the pitch deck for “the first Internet bookstore”.
  4. Panoramic Images — throw the camera in the air, get a 360×360 image from 36 2-megapixel lenses. Not sure that throwing was previously a recognised UI gesture.
Comments: 3
Four short links: 18 November 2013

Four short links: 18 November 2013

Chicago Code, 3D Smithsonian Data, AI Controlling Everything, and Game TCP

  1. The Virtuous Pipeline of Code (Public Resource) — Chicago partnering with Public Resource to open its legal codes for good. “This is great! What can we do to help?” Bravo Chicago, and everyone else—take note!
  2. Smithsonian’s 3D Data — models of 21 objects, from a gunboat to the Wright Brothers’ plane, to a wooly mammoth skeleton, to Lincoln’s life masks. I wasn’t able to find a rights statement on the site which explicitly governed the 3D models. (via Smithsonian Magazine)
  3. Anki’s Robot Cars (Xconomy) — The common characteristics of these future products, in Sofman’s mind: “Relatively simple and elegant hardware; incredibly complicated software; and Web and wireless connectivity to be able to continually expand the experience over time.” (via Slashdot)
  4. An Empirical Evaluation of TCP Performance in Online GamesWe show that because TCP was originally designed for unidirectional and network-limited bulk data transfers, it cannot adapt well to MMORPG traffic. In particular, the window-based congestion control and the fast retransmit algorithm for loss recovery are ineffective. Furthermore, TCP is overkill, as not every game packet needs to be transmitted in a reliably and orderly manner. We also show that the degraded network performance did impact users’ willingness to continue a game.
Comment

Velocity NY recap

Operating on the edge and real-time performance emerged as key themes at Velocity NY.

Maybe it was the hustle and bustle of Times Square just within earshot. Maybe it was the smell of that legendary pizza on every corner. Or maybe it was having a front row seat in the financial capital of the world while the drama of a possible government default played itself out like a Broadway show. Whatever it was, Velocity New York felt markedly different than its west coast cousin.

Of course, Velocity’s inaugural east coast incarnation sported presentations on the state of the art of all the topics we’ve come to expect from the conference: nudging style sheets, hacking Javascript, battling memory usage, tuning database queries, and coaxing the network for every last ounce of performance on the Web. (And then doing it all over again for the mobile version…)

But discussions on scaling the operations of the squishy side of the organization–developing healthy team interactions, talking through the realities of the ever-elusive DevOps culture, and what it all means to scaling not just web sites but full-fledged businesses–were more present than ever.

Read more…

Comment

Velocity: Toward the real-time business

Velocity 2013 Speaker Series

I want to start by thanking John and Steve for the warm welcome. They’ve created something very amazing with Velocity, and I’m excited to be a part of it.

It might seem a bit odd to talk about What’s Next at the beginning of a conference, but I figure the best time to go to the bank and ask for a loan is when you actually have some money.

What we’ve been talking about at Velocity, especially the DevOps side of things, is only the tip of the iceberg when it comes to how businesses are changing. And that shift is from the sequential to the concurrent. It used to be that we threw things over a series of walls, from Product Management to Design, to Development, to QA, to Production, to Customer Service and so on. That was an old world of software and one-year development cycles.

Read more…

Comment

Building for failure is a recipe for success

How you handle failure can mean the difference between "just another incident" and a revenue-stealing accident.

I was ready to get home. I’d been dozing throughout the flight from JFK to SFO, listening to the background chatter of Channel 9 as a lullaby. Somewhere over Sacramento, the rhythmic flow of controller-issued clearances and pilot confirmations was broken up by a call from our plane:

“NorCal Approach, United three-eighty-nine.”
“United three-eighty-nine, NorCal, go.”
“NorCal, United three-eighty-nine, we’d like to go ahead and…”

My headphones went silent, Channel 9 shut off.

I didn’t think too much of it as we continued our descent, flight attendants walking calmly through the cabin, getting us ready for landing. I had noticed our arrival path was one I was unfamiliar with, but nothing else seemed out of the ordinary… until we turned onto the final approach. In the turn, I noticed the unmistakable glint of firetrucks’ rotating red lights, lined up alongside the runway.

Read more…

Comment

Going beyond Onload: Measuring performance that matters

Velocity 2013 Speaker Series: Focus on Web Apps, Not Web Pages

We’re not making web pages anymore; we’re building web applications. Gone are the days of a few script tags in the <head>. Apps today are a complex web of asynchronously-loaded content and functionality. In the past decade, we’ve progressed from statically-loaded HTML to AJAX-ifying all the things. However, the way we’ve been measuring real user performance of our apps hasn’t changed to reflect our new state of art.

Defining “Done”

At what point during page load do users consider an app to be “ready enough” to start using? If we use standard performance metrics, we have to choose one of the following:

1) When the HTML document has been completely loaded and parsed, but before stylesheets, images, and subframes have finished loading (DOMContentLoaded)

2) When all synchronous scripts, stylesheets, images, and subframes have finished loading (onload)

If we pick DOMContentLoaded, it quickly becomes clear that there’s no inherent correlation between the app state at that point and what a user would consider “ready.”

Read more…

Comment: 1