"scale" entries

Four short links: 18 May 2015

Four short links: 18 May 2015

Javascript Tools, Elements of Scale, 2FA Adoption, and Empathy

  1. Tools are the Problem Tools don’t solve problems any more; they have become the problem. There’s just too many of them, and they all include an incredible number of features that you don’t use on your site –but that users are still required to download and execute.
  2. Elements of Scale: Composing and Scaling Data Platforms (Ben Stopford) — today’s data platforms range greatly in complexity, from simple caching layers or polyglotic persistence right through to wholly integrated data pipelines. There are many paths. They go to many different places. In some of these places at least, nice things are found. So, the aim for this talk is to explain how and why some of these popular approaches work. We’ll do this by first considering the building blocks from which they are composed. These are the intuitions we’ll need to pull together the bigger stuff later on.
  3. Estimating Google’s 2FA AdoptionIf we project out to the current day (965 days later), that’s a growth of ~25M users (25,586,975). Add that to the ~14M base number of users (13,886,058) exiting the graph and we end up at a grand total of…nearly 40 million users (39,473,033) enrolled in Google’s 2SV. NB there’s a lot on the back of this envelope.
  4. Empathy and Product DevelopmentNone of this means that you shouldn’t A/B test or have other quantitative measure. But all of those will mean very little if you don’t have the qualitative context that only observation and usage can provide. Empathy is central to product development.
Comment: 1
Four short links: 16 April 2015

Four short links: 16 April 2015

Relationships and Inference, Mother of All Demos, Kafka at Scale, and Real World Hardware

  1. DeepDiveDeepDive is targeted to help users extract relations between entities from data and make inferences about facts involving the entities. DeepDive can process structured, unstructured, clean, or noisy data and outputs the results into a database.
  2. From the Vault: Watching (and re-watching) “The Mother of All Demos”“I wish there was more about the social vision for computing—I worked with him for a long time, and Doug was always thinking ‘how can we collectively collaborate,’ like a sort of rock band.”
  3. Running Kafka at Scale (LinkedIn Engineering) — This tiered infrastructure solves many problems, but it greatly complicates monitoring Kafka and assuring its health. While a single Kafka cluster, when running normally, will not lose messages, the introduction of additional tiers, along with additional components such as mirror makers, creates myriad points of failure where messages can disappear. In addition to monitoring the Kafka clusters and their health, we needed to create a means to assure that all messages produced are present in each of the tiers, and make it to the critical consumers of that data.
  4. 3D Printing Titanium, and the Bin of Broken Dreams — you will learn HUGE amounts on the challenges of real-world manufacturing by reading this.
Comment
Four short links: 20 February 2015

Four short links: 20 February 2015

Robotic Garden, Kids Toys, MSFT ML, and Twitter Scale

  1. The Distributed Robotic Garden (MIT) — We consider plants, pots, and robots to be systems with different levels of mobility, sensing, actuation, and autonomy. (via Robohub)
  2. CogniToys Leverages Watson’s Brain to Befriend, Teach Your Kids (IEEE) — Through the dino, Watson’s algorithms can get to know each child that it interacts with, tailoring those interactions to the child’s age and interests.
  3. How Machine Learning Ate Microsoft (Infoworld) — Azure ML didn’t merely take the machine learning algorithms MSR had already handed over to product teams and stick them into a drag-and-drop visual designer. Microsoft has made the functionality available to developers who know the R statistical programming language and Python, which together are widely used in academic machine learning. Microsoft plans to integrate Azure ML closely with Revolution Analytics, the R startup it recently acquired.
  4. Handling Five Billion Sessions a Day in Real Time (Twitter) — infrastructure porn.
Comments: 2
Four short links: 20 January 2015

Four short links: 20 January 2015

Govt IoT, Collective Intelligence, Unknown Excellence, and Questioning Scalability

  1. Matt Webb Joining British Govt Data Service — working on IoT for them.
  2. Reading the Mind in the Eyes or Reading between the Lines? Theory of Mind Predicts Collective Intelligence (PLoS) — theory of mind abilities are a significant determinant of group collective intelligence even when, as in many online groups, the group has extremely limited communication channels. Phone/Skype calls, emails, and chats are all intensely mental activities, trying to picture the person behind the signal.
  3. MIT Faculty Search — two open gigs at MIT, one around climate change and one “undefined.” Great job ad.
  4. Scalability at What Cost?evaluation of these systems, especially in the academic context, is lacking. Folks have gotten all wound-up about scalability, despite the fact that scalability is just a means to an end (performance, capacity). When we actually look at performance, the benefits the scalable systems bring start to look much more sketchy. We’d like that to change.
Comment
Four short links: 16 December 2014

Four short links: 16 December 2014

Memory Management, Stream Processing, Robot's Google, and Emotive Words

  1. Effectively Managing Memory at Gmail Scale — how they gathered data, how Javascript memory management works, and what they did to nail down leaks.
  2. tigonan open-source, real-time, low-latency, high-throughput stream processing framework.
  3. Robo Brain — machine knowledge of the real world for robots. (via MIT Technology Review)
  4. The Structure and Interpretation of the Computer Science Curriculum — convincing argument for teaching intro to programming with Scheme, but not using the classic text SICP.

Update: the original fourth link to Depeche Mood led only to a README on GitHub; we’ve replaced it with a new link.

Comments: 5
Four short links: 12 December 2014

Four short links: 12 December 2014

Tech Ethics, Yahoo's KVS, Biology Inside, and Smart Luggage

  1. Do Artifacts Have Ethics? — 41 questions to ask yourself about the technology you create.
  2. MDBM — Yahoo’s fast key-value store, in use for over a decade. Super-fast, using mmap and passing around (gasp) raw pointers.
  3. The Revolution in Biology is Here, Now (Mike Loukides) — I’ve been asked plenty of times (and I’ve asked plenty of times), “what’s the killer product for synthetic biology?” BioFabricate convinced me that that’s the wrong question. We may never have some kind of biological iPod. That isn’t the right way to think. What I saw, instead, was real products that you might never notice. Bricks made from sand that are held together by microbes designed to excrete the binder. Bricks and packing material made from fungus (mycelium). Plastic excreted by bacteria that consume waste methane from sewage plants. You wouldn’t know, or care, whether your plastic Lego blocks are made from petroleum or from bacteria, but there’s a huge ecological difference.
  4. Bluesmart — Indiegogo campaign for a “connected carry-on,” aka a smart suitcase. From the mobile app you can track it, learn when it’s close (or too far away), (un)lock, weigh…and you can plug your devices in and recharge from the built-in battery. Sweet!
Comment
Four short links: 26 November 2014

Four short links: 26 November 2014

Metastable Failures, Static Python Analysis, Material Desktop, and AWS Scale Numbers

  1. Metastable Failure State (Facebook) — very nice story about working together to discover the cause of one of those persistently weird problems.
  2. Bandit — static security analysis of Python code.
  3. Quantum OS — Linux desktop based on Google’s Material Design. UI guidelines fascinate me: users love consistency, designers and brands hate that everything works the same.
  4. Inside AWSEvery day, AWS installs enough server infrastructure to host the entire Amazon e-tailing business from back in 2004, when Amazon the retailer was one-tenth its current size at $7 billion in annual revenue. “What has changed in the last year,” Hamilton asked rhetorically, and then quipped: “We have done it 365 more times.” That is another way of saying that in the past year AWS has added enough capacity to support a $2.55 trillion online retailing operation, should one ever be allowed to exist.
Comment
Four short links: 3 September 2014

Four short links: 3 September 2014

Distributed Systems Theory, Chinese Manufacturing, Quantified Infant, and Celebrity Data Theft

  1. Distributed Systems Theory for the Distributed Systems EngineerI tried to come up with a list of what I consider the basic concepts that are applicable to my every-day job as a distributed systems engineer; what I consider ‘table stakes’ for distributed systems engineers competent enough to design a new system.
  2. Shenzhen Trip Report (Joi Ito) — full of fascinating observations about how the balance of manufacturing strength has shifted in surprising ways. The retail price of the cheapest full featured phone is about $9. Yes. $9. This could not be designed in the US – this could only be designed by engineers with tooling grease under their fingernails who knew the manufacturing equipment inside and out, as well as the state of the art of high-end mobile phones.
  3. SproutlingThe world’s first sensing, learning, predicting baby monitor. A wearable band for your baby, a smart charger and a mobile app work together to not only monitor more effectively but learn and predict your baby’s sleep habits and optimal sleep conditions. (via Wired)
  4. Notes on the Celebrity Data Theft — wonderfully detailed analysis of how photos were lifted, and the underground industry built around them. This was one of the most unsettling aspects of these networks to me – knowing there are people out there who are turning over data on friends in their social networks in exchange for getting a dump of their private data.
Comment
Four short links: 25 August 2014

Four short links: 25 August 2014

Digital Signs, Reverse Engineering Censorship, USB Protection, and Queue Software

  1. Greenscreen — Chromecast-based open source software for digital signs.
  2. Reverse Engineering Censorship in Chinese Cyberspace (PDF) — researchers create accounts and probe to see which things are blocked. Empirical transparency.
  3. USB CondomA protective barrier between your device and “juice-jacking” hackers.
  4. queues.io — long list of job queues, message queues, and other such implementations.
Comment
Four short links: 20 August 2014

Four short links: 20 August 2014

Plant Properties, MQ Comparisons, 1915 Vis, and Mobile Web Weaknesses

  1. Machine Learning for Plant Properties — startup building database of plant genomics, properties, research, etc. for mining. The more familiar you are with your data and its meaning, the better your machine learning will be at suggesting fruitful lines of query … and the more valuable your startup will be.
  2. Dissecting Message Queues — throughput, latency, and qualitative comparison of different message queues. MQs are to modern distributed architectures what function calls were to historic unibox architectures.
  3. 1915 Data Visualization Rules — a reminder that data visualization is not new, but research into effectiveness of alternative presentation styles is.
  4. The Broken Promise of the Mobile Webit’s not just about the UI – it’s also about integration with the mobile device.
Comment