- Internet of Things: Blackett Review — the British Government’s review of Internet of Things opportunities around government. Government and others can use expert commissioning to encourage participants in demonstrator programmes to develop standards that facilitate interoperable and secure systems. Government as a large purchaser of IoT systems is going to have a big impact if it buys wisely. (via Matt Webb)
- Exactly Once Semantics with Kafka — designing for failure means it’s easier to ensure that things get done than it is to ensure that things get done exactly once.
- rdbms-subsetter — open source tool to generate a random sample of rows from a relational database that preserves referential integrity – so long as constraints are defined, all parent rows will exist for child rows. (via 18F)
- UXcheck — a browser extension to help you do a quick UX check against Nielsen’s 10 principles.
"distributed systems" entries
Four short links: 14 January 2015
IoT and Govt, Exactly Once, Random Database Subset, and UX Checking
Four short links: 24 December 2014
DRMed Objects, Eventual Consistency, Complex Systems, and Machine Learning Papers
- DRMed Cat Litter Box — the future is when you don’t own what you buy, and it’s illegal to make it work better. (via BoingBoing)
- Are We Consistent Yet? — the eventuality of consistency on different cloud platforms.
- How Complex Systems Fail (YouTube) — Richard Cook’s Velocity 2012 keynote.
- Interesting papers from NIPS 2014 — machine learning holiday reading.
Why the data center needs an operating system
It’s time for applications — not servers — to rule the data center.
Developers today are building a new class of applications. These applications no longer fit on a single server, but instead run across a fleet of servers in a data center. Examples include analytics frameworks like Apache Hadoop and Apache Spark, message brokers like Apache Kafka, key-value stores like Apache Cassandra, as well as customer-facing applications such as those run by Twitter and Netflix.
These new applications are more than applications, they are distributed systems. Just as it became commonplace for developers to build multithreaded applications for single machines, it’s now becoming commonplace for developers to build distributed systems for data centers.
But it’s difficult for developers to build distributed systems, and it’s difficult for operators to run distributed systems. Why? Because we expose the wrong level of abstraction to both developers and operators: machines. Read more…
Four short links: 28 November 2014
Material Design Inspiration, Event Processing, Launch Infrastructure, Remote Work
- Material Up — material design inspiration. MD is a physics engine for UI.
- Flafka (Cloudera) — Flume plus Kafka, offers sub-second-latency event processing without the need for dedicated infrastructure. (via Abishek Tiwari)
- terraform.io — open source package providing a common configuration to launch infrastructure, from physical and virtual servers to email and DNS providers.
- Remote Work: An Engineering Leader’s Perspective — Even proponents of remote work seem to think that you should either have a distributed team from the get go, or stick to a traditional on-site team. Our experience shows that this is incorrect…
Four short links: 26 November 2014
Metastable Failures, Static Python Analysis, Material Desktop, and AWS Scale Numbers
- Metastable Failure State (Facebook) — very nice story about working together to discover the cause of one of those persistently weird problems.
- Bandit — static security analysis of Python code.
- Quantum OS — Linux desktop based on Google’s Material Design. UI guidelines fascinate me: users love consistency, designers and brands hate that everything works the same.
- Inside AWS — Every day, AWS installs enough server infrastructure to host the entire Amazon e-tailing business from back in 2004, when Amazon the retailer was one-tenth its current size at $7 billion in annual revenue. “What has changed in the last year,” Hamilton asked rhetorically, and then quipped: “We have done it 365 more times.” That is another way of saying that in the past year AWS has added enough capacity to support a $2.55 trillion online retailing operation, should one ever be allowed to exist.
Four short links: 22 October 2014
Docker Patterns, Better Research, Streaming Framework, and Data Science Textbook
- Eight Docker Development Patterns (Vidar Hokstad) — patterns for creating repeatable builds that result in as-static-as-possible server environments.
- How to Make More Published Research True (PLOSmedicine) — overview of efforts, and research on those efforts, to raise the proportion of published research which is true.
- Gearpump — Intel’s “actor-driven streaming framework”, initial benchmarks shows that we can process 2 million messages/second (100 bytes per message) with latency around 30ms on a cluster of 4 nodes.
- Foundations of Data Science (PDF) — These notes are a first draft of a book being written by Hopcroft and Kannan [of Microsoft Research] and in many places are incomplete. However, the notes are in good enough shape to prepare lectures for a modern theoretical course in computer science.
Four short links: 14 October 2014
Science Startups, UAV Platform, Distributed vs Scalable, and Multiplayer Spreadsheet
- VCs Return to Backing Science Startups (NY Times) — industry and energy investment doubled this year, biotech up 26% in first half, but a lot of the investments are comically small and the risk remains acutely high.
- dronecode — Linux Foundation common, shared open source platform for Unmanned Aerial Vehicles (UAVs). The platform has been adopted by many of the organizations on the forefront of drone technology, including 3DRobotics, DroneDeploy, HobbyKing, Horizon Ag, PrecisionHawk, Agribotics, and Walkera, among other.
- Distributed is Not Necessarily Moe Scalable (Murat Demiras) — well-reasoned and summed up in this tweet by @jamesiry: Some people when faced with a problem think, I know, I’ll use distributed computing. Now they have N^2 problems..
- ethersheet — open source collaborative/multiplayer spreadsheet.
Four short links: 13 October 2014
Angular Style, Consensus Filters, BASE Banks, and Browser Performance
- Angular JS Style Guide — I love style guides, to the point of having posted (I think) three for Angular. Reading other people’s style guides is like listening to them make-up after arguments: you learn what’s important to them, and what they regret.
- Consensus Filters — filtering out misreads and other errors to allow all agents, or robots, in the network to arrive at the same value asymptotically by only communicating with their neighbours.
- Why Banks are BASE not ACID — Consistency it turns out is not the Holy Grail. What trumps consistency is: Auditing, Risk Management, Availability.
- perfmap — front-end performance heatmap.
Four short links: 9 October 2014
API Docs, Top Trends, Byzantine Fault Tolerance, and Devops in Practice
- dash — offline access to API documentation. Useful for those long-haul flights without wifi …
- Gartner’s Top Trends for 2015 — ubicomp, IoT, 3d printing, pervasive analytics, context, smart machines, cloud computing, software-defined everything, web-scale IT, and security. Still not the year of the Linux desktop.
- Byzantine Fault Tolerance — Wikipedia’s readable introduction to the basic challenge in distributed systems.
- Move Fast, Break Nothing (Zach Holman) — Gartner talks about “web-scale IT”, but I think the processes and tools for putting code into product (devops) are far more transformative than the technology that scales the product delivery.
Four short links: 7 October 2014
Chinese Makers, Code Projects, Distributed Data Structures, and Networked Games
- On the Maker Movement in China (Clay Shirky) — Hardware hacking hasn’t become a hot new thing in China because it never stopped being a regular old thing.
- A History of Apache Storm and Lessons Learned (Nathan Marz) — his lessons on building, promoting, releasing, maintaining, governance … all worth reading.
- Tango: Distributed Data Structures Over a Shared Log — provides developers with the abstraction of a replicated, in-memory data structure (such as a map or a tree) backed by a shared log. (via paper summary)
- Making Fast-Paced Multiplayer Networked Games is Hard (Gamasutra) — This may all sound like smoke and mirrors because that is exactly what it is – we are just maintaining the illusion the game is playing out in wall clock time even though updates are arriving from the past.