ENTRIES TAGGED "distributed systems"

Four short links: 18 August 2014

Four short links: 18 August 2014

Space Trading, Robot Capitalism, Packet Injection, and CAP Theorem

  1. Oolite — open-source clone of Elite, the classic space trading game from the 80s.
  2. Who Owns the Robots Rules The World (PDF) — interesting finding: As companies substitute machines and computers for human activity, workers need to own part of the capital stock that substitutes for them to benefit from these new “robot” technologies. Workers could own shares of the firm, hold stock options, or be paid in part from the profits. Without ownership stakes, workers will become serfs working on behalf of the robots’ overlords. Governments could tax the wealthy capital owners and redistribute income to workers, but that is not the direction societies are moving in. Workers need to own capital rather than rely on government income redistribution policies. (via Robotenomics)
  3. Schrodinger’s Cat Video and the Death of Clear-Text (Morgan Marquis-Boire) — report, based on leaked information, about use of network injection appliances targeted unencrypted pages from major providers. Compromising a target becomes as simple as waiting for the user to view unencrypted content on the Internet.
  4. CAP 12 Years Later: How the Rules Have Changed — a rundown of strategies available to deal with partitions (“outages”) in a distributed system.

Four short links: 11 August 2014

Startup Anthropology, Ends to Means, Permission to Test, and Distributed Systems Research

  1. Anthropology of Mid-Sized Startups — old but good post about the structures, norms, and dimensions of startup culture. Like a religion, a startup will care for its collective interest by defining certain things as sacred. A classic example is the company’s logo. This symbol is, quite literally, “set apart and forbidden” by brand guidelines, which often specify exactly how the logo must be presented and how far it should sit from the other elements on a page (thus separating the sacred from the profane).
  2. What Leads To — I love the elegant mechanic of decomposing an end back to a means you can do right now. Lots more sophistication obviously possible, but the fact that it’s not just about “thumbs up this end!” or about actions divorced from intention, makes it a step ahead for social software.
  3. Researching Link Rot (Pinboard) — graceful notification of a test, and with the simple ability to opt-out.
  4. The Space Between Theory and Practice in Distributed Systems (Marc Brooker) — I went through everything I’ve read on distributed systems and arranged them on a spectrum from theory to practice the two ends would be really well populated, but the middle would be disturbingly empty. Worse, changing to a graph of citation links would show a low density from theory to practice.
Four short links: 8 August 2014

Four short links: 8 August 2014

Synchronization, Security Pi, YouTube Stardom, and Javascript Logging

  1. Everything You Wanted to Know About Synchronization But Were Too Afraid to Ask (PDF) — This paper presents the most exhaustive study of synchronization to date. We span multiple layers, from hardware cache-coherence protocols up to high-level concurrent software. We do so on different types of architectures, from single-socket — uniform and non- uniform — to multi-socket — directory and broadcast-based many-cores. We draw a set of observations that, roughly speaking, imply that scalability of synchronization is mainly a property of the hardware.
  2. Raspberry Pi as Low-Cost Security Camera (Instructables) — $120 HD motion-sensing web-viewable security camera.
  3. Inside YouTube’s Fame Factory (FastCompany) — great article about the tipping point where peer-to-peer fame becomes stage-managed corporate fame, as Vidcon grows. See also Variety: If YouTube stars are swallowed by Hollywood, they are in danger of becoming less authentic versions of themselves, and teenagers will be able to pick up on that,” Sehdev says. “That could take away the one thing that makes YouTube stars so appealing.”
  4. Sherlog.js (Github) — Javascript error and event tracker application. Honestly, I have no idea if this is any good but the name is golden. I’m such a sucker.
Four short links: 16 July 2014

Four short links: 16 July 2014

Distributed Systems Design 101, Patent Trolls, Intel's Half a Billion from IoT, and Google's Project Zero.

  1. Inside bit.ly’s Distributed Systems — this is a 101 for modern web distributed systems design.
  2. Patent Trolls are Now 67% of New Patent Lawsuits in USA (WaPo) — data from PwC.
  3. Intel Made Half a Billion from Internet of Things Last Year (Quartz) — half a billion here, half a billion there, pretty soon it adds up to real money.
  4. Google’s Project Zero (Wired) — G pays a team to attack common software and report the bugs to the manufacturer. Interesting hypothesis about how the numbers inbalance between Every Russian 14 Year Old and this small team doesn’t matter: modern hacker exploits often chain together a series of hackable flaws to defeat a computer’s defenses. Kill one of those bugs and the entire exploit fails. That means Project Zero may be able to nix entire collections of exploits by finding and patching flaws in a small part of an operating system, like the “sandbox” that’s meant to limit an application’s access to the rest of the computer. ”On certain attack surfaces, we’re optimistic we can fix the bugs faster than they’re being introduced,” Hawkes says. “If you funnel your research into these limited areas, you increase the chances of bug collisions.”

From the network interface to the database

All systems are distributed systems, and we’re starting to see how they fit into Velocity's themes.


From the beginning, the Velocity Conference has focused on web performance and operations — specifically, web operations. This focus has been fairly narrow: browser performance dominated the discussion of “web performance,” and interactions between developers and IT staff dominated operations.

These limits weren’t bad. Perceived performance really is dominated by the browser — how fast you can get resources (HTML, images, CSS files, JavaScript libraries) over the network to the browser, and how fast the browser can execute those resources. How long before a user stops waiting for your page to load and clicks away? How do you make a page useable as quickly as possible, even before all the resources have loaded? Those discussions were groundbreaking and surprising: users are incredibly sensitive to page speed.

That’s not to say that Velocity hasn’t looked at the rest of the application stack; there’s been an occasional glance in the direction of the database and an even more occasional glance at the middleware. But the database and middleware have, at least historically, played a bit part. And while the focus of Velocity has been front-end tuning, speakers like Baron Schwartz haven’t let us ignore the database entirely. Read more…

Comment: 1

Beyond the stack

The tools in the Distributed Developer's Stack make development manageable in a highly distributed environment.

Cairn at Garvera, Surselva, Graubuenden, Switzerland. The shape of software development has changed radically in the last two decades. We’ve seen many changes: the Internet, the web, virtualization, and cloud computing. All of these changes point toward a fundamental new reality: all computing has become distributed computing. The age of standalone applications has disappeared, and applications that run on a single computer are almost inconceivable. Distributed is the default; and whether an application is running on Amazon Web Services (AWS), on a private cloud, or even on a desktop or a mobile phone, it depends on the behavior of other systems and services that aren’t under the developer’s control.

In the past few years, a new toolset has grown up to support the development of massively distributed applications. We call this new toolset the Distributed Developer’s Stack (DDS). It is orthogonal to the more traditional world of servers, frameworks, and operating systems; it isn’t a replacement for the aged LAMP stack, but a set of tools to make development manageable in a highly distributed environment.

The DDS is more of a meta-stack than a “stack” in the traditional sense. It’s not prescriptive; we don’t care whether you use AWS or OpenStack, whether you use Git or Mercurial. We do care that you develop for the cloud, and that you use a distributed version control system. The DDS is about the requirements for working effectively in the second decade of the 21st century. The specific tools have evolved, and will continue to evolve, and we expect you to evolve, too. Read more…


Everything is distributed

How do we manage systems that are too large to understand, too complex to control, and that fail in unpredictable ways?


“What is surprising is not that there are so many accidents. It is that there are so few. The thing that amazes you is not that your system goes down sometimes, it’s that it is up at all.”—Richard Cook

In September 2007, Jean Bookout, 76, was driving her Toyota Camry down an unfamiliar road in Oklahoma, with her friend Barbara Schwarz seated next to her on the passenger side. Suddenly, the Camry began to accelerate on its own. Bookout tried hitting the brakes, applying the emergency brake, but the car continued to accelerate. The car eventually collided with an embankment, injuring Bookout and killing Schwarz. In a subsequent legal case, lawyers for Toyota pointed to the most common of culprits in these types of accidents: human error. “Sometimes people make mistakes while driving their cars,” one of the lawyers claimed. Bookout was older, the road was unfamiliar, these tragic things happen. Read more…

Comments: 5
Four short links: 15 May 2014

Four short links: 15 May 2014

Pervasive Monitoring, Mozilla DRM, Game Finances, and Distributed Systems

  1. Pervasive Monitoring is an Attack (Tim Bray) — if your ap­pli­ca­tion doesn’t sup­port pri­va­cy, that’s prob­a­bly a bug in your ap­pli­ca­tion.
  2. Reconciling Mozilla’s Mission and the W3C EME — essentially, “we don’t want to put a closed source bolus of evil into our open source unicorn, but you won’t be able to watch House of Cards with Firefox if we don’t.”
  3. The Financial Future of Game Developers (Raph Koster) — Today, a console is really just a hardware front end to a digital publisher/distribution network/storefront. [...] Any structure that depends solely on blockbusters is not long for this world, because there is a significant component of luck in what drives popularity, so every release is literally a gamble. [...] The median game uploaded to the App Store makes zero dollars. It starts great and just gets better. Koster is on fire! He scores again! GOOOOOOOOOOOOOOOAL!
  4. Notes on Distributed Systems for Young Bloods“It’s slow” is the hardest problem you’ll ever debug.
Four short links: 3 January 2014

Four short links: 3 January 2014

Mesh Networks, Collaborative LaTeX, Distributed Systems Book, and Reverse-Engineering Netflix Metadata

  1. Commotion — open source mesh networks.
  2. WriteLaTeX — online collaborative LaTeX editor. No, really. This exists. In 2014.
  3. Distributed Systems — free book for download, goal is to bring together the ideas behind many of the more recent distributed systems – systems such as Amazon’s Dynamo, Google’s BigTable and MapReduce, Apache’s Hadoop etc.
  4. How Netflix Reverse-Engineered Hollywood (The Atlantic) — Using large teams of people specially trained to watch movies, Netflix deconstructed Hollywood. They paid people to watch films and tag them with all kinds of metadata. This process is so sophisticated and precise that taggers receive a 36-page training document that teaches them how to rate movies on their sexually suggestive content, goriness, romance levels, and even narrative elements like plot conclusiveness.
Four short links: 3 September 2013

Four short links: 3 September 2013

Fanout Architectures, In-Browser Emulation, Paean to Programmability, and Social Hardware

  1. Achieving Rapid Response Times in Large Online Services (PDF) — slides from a talk by Jeff Dean on fanout architectures. (via Alex Dong)
  2. JS MESS — porting the stunning MESS ultimate emulator to Javascript. Interesting challenges like Chrome currently has a hardcoded limit of 32,767 variables, and Firefox has a point at which it hits the CTRL-FUCKIT button and drops the mic.
  3. Go Ahead, Mess with Texas Instruments (The Atlantic) — School typically assumes that answers fall neatly into categories of “right” and “wrong.” As a conventional tool for computing “right” answers, calculators often legitimize this idea; the calculator solves problems, gives answers. But once an endorsed, conventional calculator becomes a subversive, programmable computer it destabilizes this polarity. Programming undermines the distinction between “right” and “wrong” by emphasizing the fluidity between the two. In programming, there is no “right” answer. Sure, a program might not compile or run, but making it offers multiple pathways to success, many of which are only discovered through a series of generative failures. Programming does not reify “rightness;” instead, it orients the programmer toward intentional reading, debugging, and refining of language to ensure clarity.
  4. When A Spouse Puts On Google Glass (NY Times) — Google Glass made me realize how comparably social mobile phones are. [...] People gather around phones to watch YouTube videos or look at a funny tweet together or jointly analyze a text from a friend. With Glass, there was no such sharing.