"distributed systems" entries

Four short links: 22 October 2014

Four short links: 22 October 2014

Docker Patterns, Better Research, Streaming Framework, and Data Science Textbook

  1. Eight Docker Development Patterns (Vidar Hokstad) — patterns for creating repeatable builds that result in as-static-as-possible server environments.
  2. How to Make More Published Research True (PLOSmedicine) — overview of efforts, and research on those efforts, to raise the proportion of published research which is true.
  3. Gearpump — Intel’s “actor-driven streaming framework”, initial benchmarks shows that we can process 2 million messages/second (100 bytes per message) with latency around 30ms on a cluster of 4 nodes.
  4. Foundations of Data Science (PDF) — These notes are a first draft of a book being written by Hopcroft and Kannan [of Microsoft Research] and in many places are incomplete. However, the notes are in good enough shape to prepare lectures for a modern theoretical course in computer science.
Comment
Four short links: 14 October 2014

Four short links: 14 October 2014

Science Startups, UAV Platform, Distributed vs Scalable, and Multiplayer Spreadsheet

  1. VCs Return to Backing Science Startups (NY Times) — industry and energy investment doubled this year, biotech up 26% in first half, but a lot of the investments are comically small and the risk remains acutely high.
  2. dronecode — Linux Foundation common, shared open source platform for Unmanned Aerial Vehicles (UAVs). The platform has been adopted by many of the organizations on the forefront of drone technology, including 3DRobotics, DroneDeploy, HobbyKing, Horizon Ag, PrecisionHawk, Agribotics, and Walkera, among other.
  3. Distributed is Not Necessarily Moe Scalable (Murat Demiras) — well-reasoned and summed up in this tweet by @jamesiry: Some people when faced with a problem think, I know, I’ll use distributed computing. Now they have N^2 problems..
  4. ethersheet — open source collaborative/multiplayer spreadsheet.
Comment
Four short links: 13 October 2014

Four short links: 13 October 2014

Angular Style, Consensus Filters, BASE Banks, and Browser Performance

  1. Angular JS Style Guide — I love style guides, to the point of having posted (I think) three for Angular. Reading other people’s style guides is like listening to them make-up after arguments: you learn what’s important to them, and what they regret.
  2. Consensus Filters — filtering out misreads and other errors to allow all agents, or robots, in the network to arrive at the same value asymptotically by only communicating with their neighbours.
  3. Why Banks are BASE not ACIDConsistency it turns out is not the Holy Grail. What trumps consistency is: Auditing, Risk Management, Availability.
  4. perfmap — front-end performance heatmap.
Comment
Four short links: 9 October 2014

Four short links: 9 October 2014

API Docs, Top Trends, Byzantine Fault Tolerance, and Devops in Practice

  1. dashoffline access to API documentation. Useful for those long-haul flights without wifi …
  2. Gartner’s Top Trends for 2015 — ubicomp, IoT, 3d printing, pervasive analytics, context, smart machines, cloud computing, software-defined everything, web-scale IT, and security. Still not the year of the Linux desktop.
  3. Byzantine Fault Tolerance — Wikipedia’s readable introduction to the basic challenge in distributed systems.
  4. Move Fast, Break Nothing (Zach Holman) — Gartner talks about “web-scale IT”, but I think the processes and tools for putting code into product (devops) are far more transformative than the technology that scales the product delivery.
Comment
Four short links: 7 October 2014

Four short links: 7 October 2014

Chinese Makers, Code Projects, Distributed Data Structures, and Networked Games

  1. On the Maker Movement in China (Clay Shirky) — Hardware hacking hasn’t become a hot new thing in China because it never stopped being a regular old thing.
  2. A History of Apache Storm and Lessons Learned (Nathan Marz) — his lessons on building, promoting, releasing, maintaining, governance … all worth reading.
  3. Tango: Distributed Data Structures Over a Shared Logprovides developers with the abstraction of a replicated, in-memory data structure (such as a map or a tree) backed by a shared log. (via paper summary)
  4. Making Fast-Paced Multiplayer Networked Games is Hard (Gamasutra) — This may all sound like smoke and mirrors because that is exactly what it is – we are just maintaining the illusion the game is playing out in wall clock time even though updates are arriving from the past.
Comment
Four short links: 2 October 2014

Four short links: 2 October 2014

I Heart Logs, CS50 Eating The World, Meeting Transcripts, Binary Analysis

  1. I Heart Logs — I linked to Jay Kreps’s awesome blog post twice, and now he’s expanded it into a slim O’Reilly volume which I shall press into the hands of every engineer I meet. Have you heard the Good News?
  2. CS50 Record Numbers — nearly 12% of Harvard now takes Intro to CS. (via Greg Linden)
  3. SayIt — open source from MySociety, a whole new way to organise, publish,
    and share your transcripts
    . They really want to make a better experience for sharing and organising transcripts of meetings.
  4. BAP — Binary Analysis Platform from CMU. Translates binary into assembly and then into an intermediate language which explicitly represents the side effects of assembly instructions, such as flag computations.
Comment
Four short links: 23 September 2014

Four short links: 23 September 2014

Alibubble, Reactive Manifesto, Lovely Logs, and Learning Design

  1. Thoughts on the Alibaba BubbleAlibaba is an offering that encourages the kind of negative Wall Street behavior none of us really want to see happen again. (via Tim O’Reilly)
  2. Reactive ManifestoWe believe that a coherent approach to systems architecture is needed, and we believe that all necessary aspects are already recognised individually: we want systems that are Responsive, Resilient, Elastic and Message Driven. We call these Reactive Systems.
  3. The Log: What Every Software Engineer Should Know About Real-time Data’s Unifying Abstraction (LinkedIn) — You can reduce the problem of making multiple machines all do the same thing to the problem of implementing a distributed consistent log to feed these processes input. The purpose of the log here is to squeeze all the non-determinism out of the input stream to ensure that each replica processing this input stays in sync.
  4. Method of Actiona collection of tools, games and articles to help you learn design.
Comment
Four short links: 5 September 2014

Four short links: 5 September 2014

Pragmatic Ventures?, Pictures Vanishing, Vertical Progress, and Visualising Distributed Consensus

  1. Intellectual Ventures Making Things (Bloomberg) — Having earned billions in payouts from powerful technology companies, IV is setting out to build things on its own. Rather than keeping its IP under lock and key, the company is looking to see if its ideas can be turned into products and the basis for new companies. Crazy idea. Madness. Building things never works.
  2. Twitpic Shutting Down — I guess we know what Jason Scott will be doing for the next three weeks.
  3. Thiel’s Contrarian Strategy (Fortune) — the distinction Thiel draws between transformative, “vertical” change—going from zero to one—and incremental, “horizontal” change—going from one to n. “If you take one typewriter and build 100, you have made horizontal progress,” he explains in the book’s first chapter. “If you have a typewriter and build a word processor, you have made vertical progress.”
  4. Raft: Understandable Distributed Consensus — making sense of something that’s useful but not intuitive. Awesome.
Comment
Four short links: 3 September 2014

Four short links: 3 September 2014

Distributed Systems Theory, Chinese Manufacturing, Quantified Infant, and Celebrity Data Theft

  1. Distributed Systems Theory for the Distributed Systems EngineerI tried to come up with a list of what I consider the basic concepts that are applicable to my every-day job as a distributed systems engineer; what I consider ‘table stakes’ for distributed systems engineers competent enough to design a new system.
  2. Shenzhen Trip Report (Joi Ito) — full of fascinating observations about how the balance of manufacturing strength has shifted in surprising ways. The retail price of the cheapest full featured phone is about $9. Yes. $9. This could not be designed in the US – this could only be designed by engineers with tooling grease under their fingernails who knew the manufacturing equipment inside and out, as well as the state of the art of high-end mobile phones.
  3. SproutlingThe world’s first sensing, learning, predicting baby monitor. A wearable band for your baby, a smart charger and a mobile app work together to not only monitor more effectively but learn and predict your baby’s sleep habits and optimal sleep conditions. (via Wired)
  4. Notes on the Celebrity Data Theft — wonderfully detailed analysis of how photos were lifted, and the underground industry built around them. This was one of the most unsettling aspects of these networks to me – knowing there are people out there who are turning over data on friends in their social networks in exchange for getting a dump of their private data.
Comment
Four short links: 18 August 2014

Four short links: 18 August 2014

Space Trading, Robot Capitalism, Packet Injection, and CAP Theorem

  1. Oolite — open-source clone of Elite, the classic space trading game from the 80s.
  2. Who Owns the Robots Rules The World (PDF) — interesting finding: As companies substitute machines and computers for human activity, workers need to own part of the capital stock that substitutes for them to benefit from these new “robot” technologies. Workers could own shares of the firm, hold stock options, or be paid in part from the profits. Without ownership stakes, workers will become serfs working on behalf of the robots’ overlords. Governments could tax the wealthy capital owners and redistribute income to workers, but that is not the direction societies are moving in. Workers need to own capital rather than rely on government income redistribution policies. (via Robotenomics)
  3. Schrodinger’s Cat Video and the Death of Clear-Text (Morgan Marquis-Boire) — report, based on leaked information, about use of network injection appliances targeted unencrypted pages from major providers. Compromising a target becomes as simple as waiting for the user to view unencrypted content on the Internet.
  4. CAP 12 Years Later: How the Rules Have Changed — a rundown of strategies available to deal with partitions (“outages”) in a distributed system.
Comment