"scale" entries

Four short links: 11 March 2016

Four short links: 11 March 2016

Deep-Learning Catan, Scala Tutorials, Legal Services, and Shiny Echo

  1. Strategic Dialogue Management via Deep Reinforcement Learning (Adrian Colyer) — a neural network learns to play Settlers of Catan. Is nothing sacred?
  2. scala school — Twitter’s instructional material for coming up to speed on scala.
  3. Robin Hood Fellowship — fellowship to use technology to increase access to legal services for New Yorkers. Stuff that matters.
  4. The Echo From Amazon Brims With Groundbreaking Promise (NY Times) — A bit more than a year after its release, the Echo has morphed from a gimmicky experiment into a device that brims with profound possibility. The longer I use it, the more regularly it inspires the same sense of promise I felt when I used the first iPhone — a sense this machine is opening up a vast new realm in personal computing, and gently expanding the role that computers will play in our future.
Four short links: 9 February 2016

Four short links: 9 February 2016

Collaborative Mario Agents, ElasticSearch at Scale, Anomaly Detection, Robotics Experiment

  1. Social Intelligence in Mario Bros (YouTube) — collaborative agents built by cognitive AI researchers … they have drives, communicate, learn from each other, and solve problems. Oh, and the agents are Mario, Luigi, Yoshi, and Toad within a Super Mario Brothers clone. No code or papers about it on the research group’s website yet, just a YouTube video and a press release on the university’s website, so appropriately adjust your priors for imminent world destruction at the hands of a rampaging super-AI. (via gizmag)
  2. How we Monitor and Run ElasticSearch at Scale (SignalFx) — sweet detail on metrics, dashboards, and alerting.
  3. Simple Anomaly Detection for Weekly PatternsRule-based heuristics do not scale and do not adapt easily, especially if we have thousands of alarms to set up. Some statistical approach is needed that is generic enough to handle many different metric behaviours.
  4. How to Design a Robotics Experiment (Robohub) — although there are many good experimental scientists in the robotic community, there has not been uniformly good experimental work and reporting within the community as a whole. This has advice such as “the five components of a well-designed experiment.”
Four short links: 13 October 2015

Four short links: 13 October 2015

Apple Chips, Death of the Data Center, IBM R&D, and Stateful Services

  1. Apple’s Incredible Platform Advantage (Steve Cheney) — the best people in chip design no longer want to work at Intel or Qualcomm. They want to work at Apple. I have plenty of friends in the Valley who affirm this. Sure Apple products are cooler. But Apple has also surpassed Intel in performance. This is insane. A device company – which makes CPUs for internal use – surpassing Intel, the world’s largest chip maker that practically invented the CPU and has thousands of customers.
  2. Data Center’s Days are Numbered — Adrian Cockroft says, the investments going into bolstering security on AWS and other clouds are set to pay off to the point where within five years, “it will be impossible to get security certification if you’re not running in the cloud because the tools designed for data centers are sloppily put together and can’t offer the auditing for PCI and other regulators.”
  3. A Peek Inside IBM’s R&D LabIBM still has a physics department, but at this point, almost every physicist is somehow linked to a product plan or customer plan.
  4. Building Scalable Stateful Services (High Scalability) — elucidation of a talk by Caitie McCaffrey (YouTube), tech lead for observability at Twitter.
Four short links: 29 July 2015

Four short links: 29 July 2015

Mobile Medical Scanner, Amazon Hardware Showcase, Consistency Challenges, and Govt Alpha Geeks

  1. Cellphone-Based Hand-Held Microplate Reader for Point-of-Care Testing of Enzyme-Linked Immunosorbent Assayswe created a hand-held and cost-effective cellphone-based colorimetric microplate reader that implements a routine hospital test used to identify HIV and other conditions. (via RtoZ)
  2. Amazon Launchpad — a showcase for new hardware startups, who might well be worried about Amazon’s “watch what sells and sell a generic version of it” business model.
  3. Challenges to Adopting Stronger Consistency at Scale (PDF) — It is not obvious that a system that trades stronger consistency for increased latency or reduced availability would be a net benefit to people using Facebook, especially when compared against a weakly consistent system that resolves many inconsistencies with ad hoc mechanisms.
  4. The White House’s Alpha Geeks — Megan Smith for President. I realize now there’s two things we techies should do — one is go where there are lots of us, like MIT or Silicon Valley or whatever, because you can move really fast and do extraordinary things. The other is, go where you’re rare.It’s almost like you’re a frog in boiling water; you don’t really realize how un-diverse it is until you’re in a normal diverse American innovative community like the President’s team. And then you go back and you’re like, wow. You feel, “Man, this industry is so awesome and yet we’re missing all of this talent.”
Four short links: 18 May 2015

Four short links: 18 May 2015

Javascript Tools, Elements of Scale, 2FA Adoption, and Empathy

  1. Tools are the Problem Tools don’t solve problems any more; they have become the problem. There’s just too many of them, and they all include an incredible number of features that you don’t use on your site –but that users are still required to download and execute.
  2. Elements of Scale: Composing and Scaling Data Platforms (Ben Stopford) — today’s data platforms range greatly in complexity, from simple caching layers or polyglotic persistence right through to wholly integrated data pipelines. There are many paths. They go to many different places. In some of these places at least, nice things are found. So, the aim for this talk is to explain how and why some of these popular approaches work. We’ll do this by first considering the building blocks from which they are composed. These are the intuitions we’ll need to pull together the bigger stuff later on.
  3. Estimating Google’s 2FA AdoptionIf we project out to the current day (965 days later), that’s a growth of ~25M users (25,586,975). Add that to the ~14M base number of users (13,886,058) exiting the graph and we end up at a grand total of…nearly 40 million users (39,473,033) enrolled in Google’s 2SV. NB there’s a lot on the back of this envelope.
  4. Empathy and Product DevelopmentNone of this means that you shouldn’t A/B test or have other quantitative measure. But all of those will mean very little if you don’t have the qualitative context that only observation and usage can provide. Empathy is central to product development.
Four short links: 16 April 2015

Four short links: 16 April 2015

Relationships and Inference, Mother of All Demos, Kafka at Scale, and Real World Hardware

  1. DeepDiveDeepDive is targeted to help users extract relations between entities from data and make inferences about facts involving the entities. DeepDive can process structured, unstructured, clean, or noisy data and outputs the results into a database.
  2. From the Vault: Watching (and re-watching) “The Mother of All Demos”“I wish there was more about the social vision for computing—I worked with him for a long time, and Doug was always thinking ‘how can we collectively collaborate,’ like a sort of rock band.”
  3. Running Kafka at Scale (LinkedIn Engineering) — This tiered infrastructure solves many problems, but it greatly complicates monitoring Kafka and assuring its health. While a single Kafka cluster, when running normally, will not lose messages, the introduction of additional tiers, along with additional components such as mirror makers, creates myriad points of failure where messages can disappear. In addition to monitoring the Kafka clusters and their health, we needed to create a means to assure that all messages produced are present in each of the tiers, and make it to the critical consumers of that data.
  4. 3D Printing Titanium, and the Bin of Broken Dreams — you will learn HUGE amounts on the challenges of real-world manufacturing by reading this.
Four short links: 20 February 2015

Four short links: 20 February 2015

Robotic Garden, Kids Toys, MSFT ML, and Twitter Scale

  1. The Distributed Robotic Garden (MIT) — We consider plants, pots, and robots to be systems with different levels of mobility, sensing, actuation, and autonomy. (via Robohub)
  2. CogniToys Leverages Watson’s Brain to Befriend, Teach Your Kids (IEEE) — Through the dino, Watson’s algorithms can get to know each child that it interacts with, tailoring those interactions to the child’s age and interests.
  3. How Machine Learning Ate Microsoft (Infoworld) — Azure ML didn’t merely take the machine learning algorithms MSR had already handed over to product teams and stick them into a drag-and-drop visual designer. Microsoft has made the functionality available to developers who know the R statistical programming language and Python, which together are widely used in academic machine learning. Microsoft plans to integrate Azure ML closely with Revolution Analytics, the R startup it recently acquired.
  4. Handling Five Billion Sessions a Day in Real Time (Twitter) — infrastructure porn.
Four short links: 20 January 2015

Four short links: 20 January 2015

Govt IoT, Collective Intelligence, Unknown Excellence, and Questioning Scalability

  1. Matt Webb Joining British Govt Data Service — working on IoT for them.
  2. Reading the Mind in the Eyes or Reading between the Lines? Theory of Mind Predicts Collective Intelligence (PLoS) — theory of mind abilities are a significant determinant of group collective intelligence even when, as in many online groups, the group has extremely limited communication channels. Phone/Skype calls, emails, and chats are all intensely mental activities, trying to picture the person behind the signal.
  3. MIT Faculty Search — two open gigs at MIT, one around climate change and one “undefined.” Great job ad.
  4. Scalability at What Cost?evaluation of these systems, especially in the academic context, is lacking. Folks have gotten all wound-up about scalability, despite the fact that scalability is just a means to an end (performance, capacity). When we actually look at performance, the benefits the scalable systems bring start to look much more sketchy. We’d like that to change.
Four short links: 16 December 2014

Four short links: 16 December 2014

Memory Management, Stream Processing, Robot's Google, and Emotive Words

  1. Effectively Managing Memory at Gmail Scale — how they gathered data, how Javascript memory management works, and what they did to nail down leaks.
  2. tigonan open-source, real-time, low-latency, high-throughput stream processing framework.
  3. Robo Brain — machine knowledge of the real world for robots. (via MIT Technology Review)
  4. The Structure and Interpretation of the Computer Science Curriculum — convincing argument for teaching intro to programming with Scheme, but not using the classic text SICP.

Update: the original fourth link to Depeche Mood led only to a README on GitHub; we’ve replaced it with a new link.

Four short links: 12 December 2014

Four short links: 12 December 2014

Tech Ethics, Yahoo's KVS, Biology Inside, and Smart Luggage

  1. Do Artifacts Have Ethics? — 41 questions to ask yourself about the technology you create.
  2. MDBM — Yahoo’s fast key-value store, in use for over a decade. Super-fast, using mmap and passing around (gasp) raw pointers.
  3. The Revolution in Biology is Here, Now (Mike Loukides) — I’ve been asked plenty of times (and I’ve asked plenty of times), “what’s the killer product for synthetic biology?” BioFabricate convinced me that that’s the wrong question. We may never have some kind of biological iPod. That isn’t the right way to think. What I saw, instead, was real products that you might never notice. Bricks made from sand that are held together by microbes designed to excrete the binder. Bricks and packing material made from fungus (mycelium). Plastic excreted by bacteria that consume waste methane from sewage plants. You wouldn’t know, or care, whether your plastic Lego blocks are made from petroleum or from bacteria, but there’s a huge ecological difference.
  4. Bluesmart — Indiegogo campaign for a “connected carry-on,” aka a smart suitcase. From the mobile app you can track it, learn when it’s close (or too far away), (un)lock, weigh…and you can plug your devices in and recharge from the built-in battery. Sweet!