# AT&T Fiber cuts remind us: Location is a Basket too!

The fiber cuts affecting much of the San Francisco Bay Area this week are similar to the outages in the Middle East last year (radar post), although far more limited in scope and impact.   What I said last year still holds true and is repeated below:

From an operations perspective these kinds of outages are nothing new, and underscore why having “many eggs in few baskets” is such a problem. I believe we will see similar incidents when we have the first multi-datacenter failures where multiple providers lose significant parts of their infrastructure in a single geographic area.

To really understand the issue, I recommend Neal Stephenson’s incredible (and lengthy) Wired article from 1996 entitled “Mother Earth Mother Board“:

[...] It sometimes seems as though every force of nature, every flaw in the human character, and every biological organism on the planet is engaged in a competition to see which can sever the most cables. The Museum of Submarine Telegraphy in Porthcurno, England, has a display of wrecked cables bracketed to a slab of wood. Each is labeled with its cause of failure, some of which sound dramatic, some cryptic, some both: trawler maul, spewed core, intermittent disconnection, strained core, teredo worms, crab’s nest, perished core, fish bite, even “spliced by Italians.” The teredo worm is like a science fiction creature, a bivalve with a rasp-edged shell that it uses like a buzz saw to cut through wood – or through submarine cables. Cable companies learned the hard way, early on, that it likes to eat gutta-percha, and subsequent cables received a helical wrapping of copper tape to stop it.

[...] There is also the obvious threat of sabotage by a hostile government, but, surprisingly, this almost never happens. When cypherpunk Doug Barnes was researching his Caribbean project, he spent some time looking into this, because it was exactly the kind of threat he was worried about in the case of a data haven. Somewhat to his own surprise and relief, he concluded that it simply wasn’t going to happen. “Cutting a submarine cable,” Barnes says, “is like starting a nuclear war. It’s easy to do, the results are devastating, and as soon as one country does it, all of the others will retaliate.”

As the capacity of optical fibers climbs, so does the economic damage caused when the cable is severed. FLAG makes its money by selling capacity to long-distance carriers, who turn around and resell it to end users at rates that are increasingly determined by what the market will bear. If FLAG gets chopped, no calls get through. The carriers’ phone calls get routed to FLAG’s competitors (other cables or satellites), and FLAG loses the revenue represented by those calls until the cable is repaired. The amount of revenue it loses is a function of how many calls the cable is physically capable of carrying, how close to capacity the cable is running, and what prices the market will bear for calls on the broken cable segment. In other words, a break between Dubai and Bombay might cost FLAG more in revenue loss than a break between Korea and Japan if calls between Dubai and Bombay cost more.

The rule of thumb for calculating revenue loss works like this: for every penny per minute that the long distance market will bear on a particular route, the loss of revenue, should FLAG be severed on that route, is about $3,000 a minute. So if calls on that route are a dime a minute, the damage is$30,000 a minute, and if calls are a dollar a minute, the damage is almost a third of a million dollars for every minute the cable is down. Upcoming advances in fiber bandwidth may push this figure, for some cables, past the million-dollar-a-minute mark.

It’s also worth mentioning the outages to multiple service providers hosted in a single colocation facility when the FBI sized all the equipment in the facility, the big outage at 365 Main from two years ago, and many others (see: Radar posts & comprehensive coverage at Data Center Knowledge).

(If Web Operations & Infrastructure is your interest or passion, you should attend Velocity 2009.  You can use the code “vel09cmb for a 15% discount)

(Image source: http://www.flickr.com/photos/mundane_joy/2301368102/)
• http://basiscraft.com Thomas Lord

Speaking of too many eggs and too few baskets:

How does this relate to emerging plans for the “smart grid” electrical system? One of the magnificent (and therefore quite rare) steps towards resilience that happened when the electric grid and phone networks were built out was their separation: lights go out but perhaps the phone still works, and vice versa. The indications I’m seeing, lately, are of plans to make a (so-called) “smart” grid that relies on the Internet to function properly….

-t