ENTRIES TAGGED "distributed systems"

Four short links: 23 September 2014

Four short links: 23 September 2014

Alibubble, Reactive Manifesto, Lovely Logs, and Learning Design

  1. Thoughts on the Alibaba BubbleAlibaba is an offering that encourages the kind of negative Wall Street behavior none of us really want to see happen again. (via Tim O’Reilly)
  2. Reactive ManifestoWe believe that a coherent approach to systems architecture is needed, and we believe that all necessary aspects are already recognised individually: we want systems that are Responsive, Resilient, Elastic and Message Driven. We call these Reactive Systems.
  3. The Log: What Every Software Engineer Should Know About Real-time Data’s Unifying Abstraction (LinkedIn) — You can reduce the problem of making multiple machines all do the same thing to the problem of implementing a distributed consistent log to feed these processes input. The purpose of the log here is to squeeze all the non-determinism out of the input stream to ensure that each replica processing this input stays in sync.
  4. Method of Actiona collection of tools, games and articles to help you learn design.
Four short links: 5 September 2014

Four short links: 5 September 2014

Pragmatic Ventures?, Pictures Vanishing, Vertical Progress, and Visualising Distributed Consensus

  1. Intellectual Ventures Making Things (Bloomberg) — Having earned billions in payouts from powerful technology companies, IV is setting out to build things on its own. Rather than keeping its IP under lock and key, the company is looking to see if its ideas can be turned into products and the basis for new companies. Crazy idea. Madness. Building things never works.
  2. Twitpic Shutting Down — I guess we know what Jason Scott will be doing for the next three weeks.
  3. Thiel’s Contrarian Strategy (Fortune) — the distinction Thiel draws between transformative, “vertical” change—going from zero to one—and incremental, “horizontal” change—going from one to n. “If you take one typewriter and build 100, you have made horizontal progress,” he explains in the book’s first chapter. “If you have a typewriter and build a word processor, you have made vertical progress.”
  4. Raft: Understandable Distributed Consensus — making sense of something that’s useful but not intuitive. Awesome.
Four short links: 3 September 2014

Four short links: 3 September 2014

Distributed Systems Theory, Chinese Manufacturing, Quantified Infant, and Celebrity Data Theft

  1. Distributed Systems Theory for the Distributed Systems EngineerI tried to come up with a list of what I consider the basic concepts that are applicable to my every-day job as a distributed systems engineer; what I consider ‘table stakes’ for distributed systems engineers competent enough to design a new system.
  2. Shenzhen Trip Report (Joi Ito) — full of fascinating observations about how the balance of manufacturing strength has shifted in surprising ways. The retail price of the cheapest full featured phone is about $9. Yes. $9. This could not be designed in the US – this could only be designed by engineers with tooling grease under their fingernails who knew the manufacturing equipment inside and out, as well as the state of the art of high-end mobile phones.
  3. SproutlingThe world’s first sensing, learning, predicting baby monitor. A wearable band for your baby, a smart charger and a mobile app work together to not only monitor more effectively but learn and predict your baby’s sleep habits and optimal sleep conditions. (via Wired)
  4. Notes on the Celebrity Data Theft — wonderfully detailed analysis of how photos were lifted, and the underground industry built around them. This was one of the most unsettling aspects of these networks to me – knowing there are people out there who are turning over data on friends in their social networks in exchange for getting a dump of their private data.
Four short links: 18 August 2014

Four short links: 18 August 2014

Space Trading, Robot Capitalism, Packet Injection, and CAP Theorem

  1. Oolite — open-source clone of Elite, the classic space trading game from the 80s.
  2. Who Owns the Robots Rules The World (PDF) — interesting finding: As companies substitute machines and computers for human activity, workers need to own part of the capital stock that substitutes for them to benefit from these new “robot” technologies. Workers could own shares of the firm, hold stock options, or be paid in part from the profits. Without ownership stakes, workers will become serfs working on behalf of the robots’ overlords. Governments could tax the wealthy capital owners and redistribute income to workers, but that is not the direction societies are moving in. Workers need to own capital rather than rely on government income redistribution policies. (via Robotenomics)
  3. Schrodinger’s Cat Video and the Death of Clear-Text (Morgan Marquis-Boire) — report, based on leaked information, about use of network injection appliances targeted unencrypted pages from major providers. Compromising a target becomes as simple as waiting for the user to view unencrypted content on the Internet.
  4. CAP 12 Years Later: How the Rules Have Changed — a rundown of strategies available to deal with partitions (“outages”) in a distributed system.

Four short links: 11 August 2014

Startup Anthropology, Ends to Means, Permission to Test, and Distributed Systems Research

  1. Anthropology of Mid-Sized Startups — old but good post about the structures, norms, and dimensions of startup culture. Like a religion, a startup will care for its collective interest by defining certain things as sacred. A classic example is the company’s logo. This symbol is, quite literally, “set apart and forbidden” by brand guidelines, which often specify exactly how the logo must be presented and how far it should sit from the other elements on a page (thus separating the sacred from the profane).
  2. What Leads To — I love the elegant mechanic of decomposing an end back to a means you can do right now. Lots more sophistication obviously possible, but the fact that it’s not just about “thumbs up this end!” or about actions divorced from intention, makes it a step ahead for social software.
  3. Researching Link Rot (Pinboard) — graceful notification of a test, and with the simple ability to opt-out.
  4. The Space Between Theory and Practice in Distributed Systems (Marc Brooker) — I went through everything I’ve read on distributed systems and arranged them on a spectrum from theory to practice the two ends would be really well populated, but the middle would be disturbingly empty. Worse, changing to a graph of citation links would show a low density from theory to practice.
Four short links: 8 August 2014

Four short links: 8 August 2014

Synchronization, Security Pi, YouTube Stardom, and Javascript Logging

  1. Everything You Wanted to Know About Synchronization But Were Too Afraid to Ask (PDF) — This paper presents the most exhaustive study of synchronization to date. We span multiple layers, from hardware cache-coherence protocols up to high-level concurrent software. We do so on different types of architectures, from single-socket — uniform and non- uniform — to multi-socket — directory and broadcast-based many-cores. We draw a set of observations that, roughly speaking, imply that scalability of synchronization is mainly a property of the hardware.
  2. Raspberry Pi as Low-Cost Security Camera (Instructables) — $120 HD motion-sensing web-viewable security camera.
  3. Inside YouTube’s Fame Factory (FastCompany) — great article about the tipping point where peer-to-peer fame becomes stage-managed corporate fame, as Vidcon grows. See also Variety: If YouTube stars are swallowed by Hollywood, they are in danger of becoming less authentic versions of themselves, and teenagers will be able to pick up on that,” Sehdev says. “That could take away the one thing that makes YouTube stars so appealing.”
  4. Sherlog.js (Github) — Javascript error and event tracker application. Honestly, I have no idea if this is any good but the name is golden. I’m such a sucker.
Four short links: 16 July 2014

Four short links: 16 July 2014

Distributed Systems Design 101, Patent Trolls, Intel's Half a Billion from IoT, and Google's Project Zero.

  1. Inside bit.ly’s Distributed Systems — this is a 101 for modern web distributed systems design.
  2. Patent Trolls are Now 67% of New Patent Lawsuits in USA (WaPo) — data from PwC.
  3. Intel Made Half a Billion from Internet of Things Last Year (Quartz) — half a billion here, half a billion there, pretty soon it adds up to real money.
  4. Google’s Project Zero (Wired) — G pays a team to attack common software and report the bugs to the manufacturer. Interesting hypothesis about how the numbers inbalance between Every Russian 14 Year Old and this small team doesn’t matter: modern hacker exploits often chain together a series of hackable flaws to defeat a computer’s defenses. Kill one of those bugs and the entire exploit fails. That means Project Zero may be able to nix entire collections of exploits by finding and patching flaws in a small part of an operating system, like the “sandbox” that’s meant to limit an application’s access to the rest of the computer. ”On certain attack surfaces, we’re optimistic we can fix the bugs faster than they’re being introduced,” Hawkes says. “If you funnel your research into these limited areas, you increase the chances of bug collisions.”

From the network interface to the database

All systems are distributed systems, and we’re starting to see how they fit into Velocity's themes.


From the beginning, the Velocity Conference has focused on web performance and operations — specifically, web operations. This focus has been fairly narrow: browser performance dominated the discussion of “web performance,” and interactions between developers and IT staff dominated operations.

These limits weren’t bad. Perceived performance really is dominated by the browser — how fast you can get resources (HTML, images, CSS files, JavaScript libraries) over the network to the browser, and how fast the browser can execute those resources. How long before a user stops waiting for your page to load and clicks away? How do you make a page useable as quickly as possible, even before all the resources have loaded? Those discussions were groundbreaking and surprising: users are incredibly sensitive to page speed.

That’s not to say that Velocity hasn’t looked at the rest of the application stack; there’s been an occasional glance in the direction of the database and an even more occasional glance at the middleware. But the database and middleware have, at least historically, played a bit part. And while the focus of Velocity has been front-end tuning, speakers like Baron Schwartz haven’t let us ignore the database entirely. Read more…

Comment: 1

Beyond the stack

The tools in the Distributed Developer's Stack make development manageable in a highly distributed environment.

Cairn at Garvera, Surselva, Graubuenden, Switzerland. The shape of software development has changed radically in the last two decades. We’ve seen many changes: the Internet, the web, virtualization, and cloud computing. All of these changes point toward a fundamental new reality: all computing has become distributed computing. The age of standalone applications has disappeared, and applications that run on a single computer are almost inconceivable. Distributed is the default; and whether an application is running on Amazon Web Services (AWS), on a private cloud, or even on a desktop or a mobile phone, it depends on the behavior of other systems and services that aren’t under the developer’s control.

In the past few years, a new toolset has grown up to support the development of massively distributed applications. We call this new toolset the Distributed Developer’s Stack (DDS). It is orthogonal to the more traditional world of servers, frameworks, and operating systems; it isn’t a replacement for the aged LAMP stack, but a set of tools to make development manageable in a highly distributed environment.

The DDS is more of a meta-stack than a “stack” in the traditional sense. It’s not prescriptive; we don’t care whether you use AWS or OpenStack, whether you use Git or Mercurial. We do care that you develop for the cloud, and that you use a distributed version control system. The DDS is about the requirements for working effectively in the second decade of the 21st century. The specific tools have evolved, and will continue to evolve, and we expect you to evolve, too. Read more…


Everything is distributed

How do we manage systems that are too large to understand, too complex to control, and that fail in unpredictable ways?


“What is surprising is not that there are so many accidents. It is that there are so few. The thing that amazes you is not that your system goes down sometimes, it’s that it is up at all.”—Richard Cook

In September 2007, Jean Bookout, 76, was driving her Toyota Camry down an unfamiliar road in Oklahoma, with her friend Barbara Schwarz seated next to her on the passenger side. Suddenly, the Camry began to accelerate on its own. Bookout tried hitting the brakes, applying the emergency brake, but the car continued to accelerate. The car eventually collided with an embankment, injuring Bookout and killing Schwarz. In a subsequent legal case, lawyers for Toyota pointed to the most common of culprits in these types of accidents: human error. “Sometimes people make mistakes while driving their cars,” one of the lawyers claimed. Bookout was older, the road was unfamiliar, these tragic things happen. Read more…

Comments: 5