ENTRIES TAGGED "distributed systems"

Four short links: 16 July 2014

Four short links: 16 July 2014

Distributed Systems Design 101, Patent Trolls, Intel's Half a Billion from IoT, and Google's Project Zero.

  1. Inside bit.ly’s Distributed Systems — this is a 101 for modern web distributed systems design.
  2. Patent Trolls are Now 67% of New Patent Lawsuits in USA (WaPo) — data from PwC.
  3. Intel Made Half a Billion from Internet of Things Last Year (Quartz) — half a billion here, half a billion there, pretty soon it adds up to real money.
  4. Google’s Project Zero (Wired) — G pays a team to attack common software and report the bugs to the manufacturer. Interesting hypothesis about how the numbers inbalance between Every Russian 14 Year Old and this small team doesn’t matter: modern hacker exploits often chain together a series of hackable flaws to defeat a computer’s defenses. Kill one of those bugs and the entire exploit fails. That means Project Zero may be able to nix entire collections of exploits by finding and patching flaws in a small part of an operating system, like the “sandbox” that’s meant to limit an application’s access to the rest of the computer. ”On certain attack surfaces, we’re optimistic we can fix the bugs faster than they’re being introduced,” Hawkes says. “If you funnel your research into these limited areas, you increase the chances of bug collisions.”

From the network interface to the database

All systems are distributed systems, and we’re starting to see how they fit into Velocity's themes.


From the beginning, the Velocity Conference has focused on web performance and operations — specifically, web operations. This focus has been fairly narrow: browser performance dominated the discussion of “web performance,” and interactions between developers and IT staff dominated operations.

These limits weren’t bad. Perceived performance really is dominated by the browser — how fast you can get resources (HTML, images, CSS files, JavaScript libraries) over the network to the browser, and how fast the browser can execute those resources. How long before a user stops waiting for your page to load and clicks away? How do you make a page useable as quickly as possible, even before all the resources have loaded? Those discussions were groundbreaking and surprising: users are incredibly sensitive to page speed.

That’s not to say that Velocity hasn’t looked at the rest of the application stack; there’s been an occasional glance in the direction of the database and an even more occasional glance at the middleware. But the database and middleware have, at least historically, played a bit part. And while the focus of Velocity has been front-end tuning, speakers like Baron Schwartz haven’t let us ignore the database entirely. Read more…

Comment: 1

Beyond the stack

The tools in the Distributed Developer's Stack make development manageable in a highly distributed environment.

Cairn at Garvera, Surselva, Graubuenden, Switzerland. The shape of software development has changed radically in the last two decades. We’ve seen many changes: the Internet, the web, virtualization, and cloud computing. All of these changes point toward a fundamental new reality: all computing has become distributed computing. The age of standalone applications has disappeared, and applications that run on a single computer are almost inconceivable. Distributed is the default; and whether an application is running on Amazon Web Services (AWS), on a private cloud, or even on a desktop or a mobile phone, it depends on the behavior of other systems and services that aren’t under the developer’s control.

In the past few years, a new toolset has grown up to support the development of massively distributed applications. We call this new toolset the Distributed Developer’s Stack (DDS). It is orthogonal to the more traditional world of servers, frameworks, and operating systems; it isn’t a replacement for the aged LAMP stack, but a set of tools to make development manageable in a highly distributed environment.

The DDS is more of a meta-stack than a “stack” in the traditional sense. It’s not prescriptive; we don’t care whether you use AWS or OpenStack, whether you use Git or Mercurial. We do care that you develop for the cloud, and that you use a distributed version control system. The DDS is about the requirements for working effectively in the second decade of the 21st century. The specific tools have evolved, and will continue to evolve, and we expect you to evolve, too. Read more…


Everything is distributed

How do we manage systems that are too large to understand, too complex to control, and that fail in unpredictable ways?


“What is surprising is not that there are so many accidents. It is that there are so few. The thing that amazes you is not that your system goes down sometimes, it’s that it is up at all.”—Richard Cook

In September 2007, Jean Bookout, 76, was driving her Toyota Camry down an unfamiliar road in Oklahoma, with her friend Barbara Schwarz seated next to her on the passenger side. Suddenly, the Camry began to accelerate on its own. Bookout tried hitting the brakes, applying the emergency brake, but the car continued to accelerate. The car eventually collided with an embankment, injuring Bookout and killing Schwarz. In a subsequent legal case, lawyers for Toyota pointed to the most common of culprits in these types of accidents: human error. “Sometimes people make mistakes while driving their cars,” one of the lawyers claimed. Bookout was older, the road was unfamiliar, these tragic things happen. Read more…

Comments: 5
Four short links: 15 May 2014

Four short links: 15 May 2014

Pervasive Monitoring, Mozilla DRM, Game Finances, and Distributed Systems

  1. Pervasive Monitoring is an Attack (Tim Bray) — if your ap­pli­ca­tion doesn’t sup­port pri­va­cy, that’s prob­a­bly a bug in your ap­pli­ca­tion.
  2. Reconciling Mozilla’s Mission and the W3C EME — essentially, “we don’t want to put a closed source bolus of evil into our open source unicorn, but you won’t be able to watch House of Cards with Firefox if we don’t.”
  3. The Financial Future of Game Developers (Raph Koster) — Today, a console is really just a hardware front end to a digital publisher/distribution network/storefront. [...] Any structure that depends solely on blockbusters is not long for this world, because there is a significant component of luck in what drives popularity, so every release is literally a gamble. [...] The median game uploaded to the App Store makes zero dollars. It starts great and just gets better. Koster is on fire! He scores again! GOOOOOOOOOOOOOOOAL!
  4. Notes on Distributed Systems for Young Bloods“It’s slow” is the hardest problem you’ll ever debug.
Four short links: 3 January 2014

Four short links: 3 January 2014

Mesh Networks, Collaborative LaTeX, Distributed Systems Book, and Reverse-Engineering Netflix Metadata

  1. Commotion — open source mesh networks.
  2. WriteLaTeX — online collaborative LaTeX editor. No, really. This exists. In 2014.
  3. Distributed Systems — free book for download, goal is to bring together the ideas behind many of the more recent distributed systems – systems such as Amazon’s Dynamo, Google’s BigTable and MapReduce, Apache’s Hadoop etc.
  4. How Netflix Reverse-Engineered Hollywood (The Atlantic) — Using large teams of people specially trained to watch movies, Netflix deconstructed Hollywood. They paid people to watch films and tag them with all kinds of metadata. This process is so sophisticated and precise that taggers receive a 36-page training document that teaches them how to rate movies on their sexually suggestive content, goriness, romance levels, and even narrative elements like plot conclusiveness.
Four short links: 3 September 2013

Four short links: 3 September 2013

Fanout Architectures, In-Browser Emulation, Paean to Programmability, and Social Hardware

  1. Achieving Rapid Response Times in Large Online Services (PDF) — slides from a talk by Jeff Dean on fanout architectures. (via Alex Dong)
  2. JS MESS — porting the stunning MESS ultimate emulator to Javascript. Interesting challenges like Chrome currently has a hardcoded limit of 32,767 variables, and Firefox has a point at which it hits the CTRL-FUCKIT button and drops the mic.
  3. Go Ahead, Mess with Texas Instruments (The Atlantic) — School typically assumes that answers fall neatly into categories of “right” and “wrong.” As a conventional tool for computing “right” answers, calculators often legitimize this idea; the calculator solves problems, gives answers. But once an endorsed, conventional calculator becomes a subversive, programmable computer it destabilizes this polarity. Programming undermines the distinction between “right” and “wrong” by emphasizing the fluidity between the two. In programming, there is no “right” answer. Sure, a program might not compile or run, but making it offers multiple pathways to success, many of which are only discovered through a series of generative failures. Programming does not reify “rightness;” instead, it orients the programmer toward intentional reading, debugging, and refining of language to ensure clarity.
  4. When A Spouse Puts On Google Glass (NY Times) — Google Glass made me realize how comparably social mobile phones are. [...] People gather around phones to watch YouTube videos or look at a funny tweet together or jointly analyze a text from a friend. With Glass, there was no such sharing.
Four short links: 5 June 2013

Four short links: 5 June 2013

Interesting Themes, Distributed Systems Failure Modes, Gesture Sensing Through Wifi, and Bad Taste Agile

  1. OATV Fund III Pitch Deck (Slideshare) — contains a list of what they were investing in, and what they want to invest in with the new round. Then: Quantified self; Internet subsystems; Smart networks of things; Manipulation and visualization of big data; sustainability; Maker movement. Now: Quantified Self Pro; Maker Pro; Hacking Education; Hidden Economies; Operations as Competitive Advantage; A Router in Every Pocket; The Internet Operating System. The move to “Pro” interests me, too. (via Bryce Roberts)
  2. The Network is ReliableMany applications silently degrade when the network fails, and resulting problems may not be understood for some time—if they are understood at all. [...] much of what we know about the failure modes of real-world distributed systems is founded on guesswork and rumor. [...] In this post, we’d like to bring a few of these stories together. We believe this is a first step towards a more open and honest discussion of real-world partition behavior, and, ultimately, more robust distributed systems design.
  3. Wisee (PDF) — recognising gestures using disturbances in the (wifi) force. Our results show that WiSee can identify and classify a set of nine gestures with an average accuracy of 94%. (via BoingBoing)
  4. Why Your Users Hate Agile Development (IT World) — What developers see as iterative and flexible, users see as disorganized and never-ending. Here’s how some experienced developers have changed that perception. (via Slashdot)