"distributed systems" entries

Four short links: 26 March 2015

Four short links: 26 March 2015

GPU Graph Algorithms, Data Sharing, Build Like Google, and Distributed Systems Theory

  1. gunrocka CUDA library for graph primitives that refactors, integrates, and generalizes best-of-class GPU implementations of breadth-first search, connected components, and betweenness centrality into a unified code base useful for future development of high-performance GPU graph primitives. (via Ben Lorica)
  2. How to Share Data with a Statisticiansome instruction on the best way to share data to avoid the most common pitfalls and sources of delay in the transition from data collection to data analysis.
  3. Bazela build tool, i.e. a tool that will run compilers and tests to assemble your software, similar to Make, Ant, Gradle, Buck, Pants, and Maven. Google’s build tool, to be precise.
  4. You Can’t Have Exactly-Once Delivery — not about the worst post office ever. FLP and the Two Generals Problem are not design complexities, they are impossibility results.
Comment
Four short links: 16 March 2015

Four short links: 16 March 2015

Turing and Bechdel, Women in Leadership, Coding Messaging, and VR Affordance

  1. Ex Machine: When Turing Meets the Bechdel TestThe Bechdel Test is useful for more than examining gender representation. It can be our Turing Test for creating believable alien or artificial life forms. If you look at our filmic or cultural representations of the other or alien, then you have to be struck by the singular nature of them all. From Frankenstein to Big Hero 6, do they have any reality without the central human characters? No, they are alone. Even Alien is alone. At least in Frankenstein, it is the utter aloneness of the new form that is the whole story. Films that have pushed the envelope are few. And doing a quick mental check, one was left feeling empathy for the ‘others’ in only a couple, like Westworld, BladeRunner, and Planet of the Apes, and the books of writers like Brin and Cherryh.
  2. Women in Business: The Path to Leadership (PDF) — Grant Thornton International Business Report on women in senior roles, operational and governance. Ends with some sound recommendations (via Rowan Simpson).
  3. Adventures in Messaging Queues — extremely readable tale of developing a custom piece of distributed infrastructure.
  4. Valve’s SteamVRWhat these two experiences shared was an overpowering sense of intimacy with the virtual space. This is brought home again and again in the demos, and its impact is profound. The smallness of the space you can move around – 15 feet square – is almost a virtue in this respect, because it focuses you on your immediate surroundings in a way the vast majority of video games don’t, and which has much more in common with real life.
Comment
Four short links: 12 March 2015

Four short links: 12 March 2015

Billion Node Graphs, Asynchronous Systems, Deep Learning Hardware, and Vision Resources

  1. Mining Billion Node Graphs: Patterns and Scalable Algorithms (PDF) — slides from a CMU academic’s talk at C-BIG 2012.
  2. There Is No NowOne of the most important results in the theory of distributed systems is an impossibility result, showing one of the limits of the ability to build systems that work in a world where things can fail. This is generally referred to as the FLP result, named for its authors, Fischer, Lynch, and Paterson. Their work, which won the 2001 Dijkstra Prize for the most influential paper in distributed computing, showed conclusively that some computational problems that are achievable in a “synchronous” model in which hosts have identical or shared clocks are impossible under a weaker, asynchronous system model.
  3. Deep Learning Hardware GuideOne of the worst things you can do when building a deep learning system is to waste money on hardware that is unnecessary. Here I will guide you step by step through the hardware you will need for a cheap high performance system.
  4. Awesome Computer Vision — curated list of computer vision resources.
Comment
Four short links: 10 March 2015

Four short links: 10 March 2015

Robot Swarms, Media Hacking, Inside-Out Databases, and Quantified Medical Self

  1. Surgical Micro-Robot SwarmsA swarm of medical microrobots. Start with cm sized robots. These already exist in the form of pillbots and I reference the work of Paolo Dario’s lab in this direction. Then get 10 times smaller to mm sized robots. Here we’re at the limit of making robots with conventional mechatronics. The almost successful I-SWARM project prototyped remarkable robots measuring 4 x 4 x 3mm. But now shrink by another 3 orders of magnitude to microbots, measured in micrometers. This is how small robots would have to be in order to swim through and access (most of) the vascular system. Here we are far beyond conventional materials and electronics, but amazingly work is going on to control bacteria. In the example I give from the lab of Sylvain Martel, swarms of magnetotactic bacteria are steered by an external magnetic field and, interestingly, tracked in an MRI scanner.
  2. Media Hacking — interesting discussion of the techniques used to spread disinformation through social media, often using bots to surface/promote a message.
  3. Turning the Database Inside Out with Apache Samzareplication, secondary indexing, caching, and materialized views as a way of getting into distributed stream processing.
  4. Apple Research Kit — Apple positioning their mobile personal biodata tools with medical legitimacy, presumably as a way to distance themselves from the stereotypical quantified selfer. I’m reminded of the gym chain owner who told me, about the Nike+, “yeah, maybe 5% of my clients will want this. The rest go to the gym so they can eat and drink what they want.”
Comment
Four short links: 20 February 2015

Four short links: 20 February 2015

Robotic Garden, Kids Toys, MSFT ML, and Twitter Scale

  1. The Distributed Robotic Garden (MIT) — We consider plants, pots, and robots to be systems with different levels of mobility, sensing, actuation, and autonomy. (via Robohub)
  2. CogniToys Leverages Watson’s Brain to Befriend, Teach Your Kids (IEEE) — Through the dino, Watson’s algorithms can get to know each child that it interacts with, tailoring those interactions to the child’s age and interests.
  3. How Machine Learning Ate Microsoft (Infoworld) — Azure ML didn’t merely take the machine learning algorithms MSR had already handed over to product teams and stick them into a drag-and-drop visual designer. Microsoft has made the functionality available to developers who know the R statistical programming language and Python, which together are widely used in academic machine learning. Microsoft plans to integrate Azure ML closely with Revolution Analytics, the R startup it recently acquired.
  4. Handling Five Billion Sessions a Day in Real Time (Twitter) — infrastructure porn.
Comments: 2

Variations in event-driven architecture

Find out if mediator or broker topology is right for you.

Editor’s note: this is an advance excerpt from Chapter 2 of the forthcoming Software Architecture Patterns by Mark Richards. This report looks at the patterns that define the basic characteristics and behavior of highly scalable and highly agile applications, and will be made available to download in advance of our Software Architecture Conference happening March 16-19 in Boston.

Coming soon.

Available now!

The event-driven architecture pattern is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. It is also highly adaptable and can be used for both small and large, complex applications. The pattern is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events. 

The event-driven architecture pattern consists of two main topologies, the mediator and the broker. The mediator topology is commonly used when you need to orchestrate multiple steps within an event through a central mediator, whereas the broker topology is used when you want to chain events together without the use of a central mediator. Because the architecture characteristics and implementation strategies differ between these two topologies, it is important to understand each one to know which is best suited for your particular situation.

Read more…

Comment: 1
Four short links: 9 February 2015

Four short links: 9 February 2015

iBeacon at Scale, Product Teams, Progress Bars, and Distributed Fallacies

  1. The Realities of Installing iBeacon at Scale (Brooklyn Museum) — death by a thousand mundane little real-world pains.
  2. How We Build Software (Intercom) — rare to see descriptions of how to build product teams.
  3. MProgress.js — Material Design progress bars for the web.
  4. Eight Fallacies of Distributed Computing — your network is unreliable, slow, congested, insecure, changing, expensive, and inconsistent. And that’s on a good day.
Comment
Four short links: 4 February, 2015

Four short links: 4 February, 2015

CLR Open Source, Cluster Management, Workplace Bias, and So Much Chrome

  1. CoreCLR Open Sourced (MSDN) — garbage collection, compilation to machine code, and other bits of the CLR.
  2. Kafka Manager — Yahoo open sources a cluster management tool.
  3. Unconscious Bias at Work — useful talk on workplace biases.
  4. All the Chrome (Quirksmode) — Google Chrome is not the default browser on Android 4.3+. There are now at least eight Chromium-based Android default browsers, and they are all subtly, though not wildly, different. The number of Chromium family members has recently risen from nine to eleven with the addition of HTC and LG Chromium, default browsers for modern HTC and LG high-end devices.
Comment
Four short links: 28 January 2015

Four short links: 28 January 2015

Note and Vote, Gaming Behaviour, Code Search, and Immutabilate All The Things

  1. Note and Vote (Google Ventures) — nifty meeting hack to surface ideas and identify popular candidates to a decision maker.
  2. Applying Psychology to Improve Online Behaviour — online game runs massive experiments (w/researchers to validate findings) to improve the behaviour of their players. Some of Riot’s experiments are causing the game to evolve. For example, one product is a restricted chat mode that limits the number of messages abusive players can type per match. It’s a temporary punishment that has led to a noticeable improvement in player behavior afterward —on average, individuals who went through a period of restricted chat saw 20 percent fewer abuse reports filed by other players. The restricted chat approach also proved 4 percent more effective at improving player behavior than the usual punishment method of temporarily banning toxic players. Even the smallest improvements in player behavior can make a huge difference in an online game that attracts 67 million players every month.
  3. Hound — open source code search tool from Etsy.
  4. Immutability Changes Everything (PDF) — This paper is simply an amuse-bouche on the repeated patterns of computing that leverage immutability.
Comment
Four short links: 14 January 2015

Four short links: 14 January 2015

IoT and Govt, Exactly Once, Random Database Subset, and UX Checking

  1. Internet of Things: Blackett Review — the British Government’s review of Internet of Things opportunities around government. Government and others can use expert commissioning to encourage participants in demonstrator programmes to develop standards that facilitate interoperable and secure systems. Government as a large purchaser of IoT systems is going to have a big impact if it buys wisely. (via Matt Webb)
  2. Exactly Once Semantics with Kafka — designing for failure means it’s easier to ensure that things get done than it is to ensure that things get done exactly once.
  3. rdbms-subsetter — open source tool to generate a random sample of rows from a relational database that preserves referential integrity – so long as constraints are defined, all parent rows will exist for child rows. (via 18F)
  4. UXcheck — a browser extension to help you do a quick UX check against Nielsen’s 10 principles.
Comment