"programming" entries

Four short links: 15 April 2015

Four short links: 15 April 2015

Facebook as Biometrics, Time Series Sequences, Programming Languages, and Oceanic Robots

  1. Facebook Biometrics Cache (Business Insider) — Facebook has been accused of violating the privacy of its users by collecting their facial data, according to a class-action lawsuit filed last week. This data-collection program led to its well-known automatic face-tagging service. But it also helped Facebook create “the largest privately held stash of biometric face-recognition data in the world,” the Courthouse News Service reports.
  2. The Clustering of Time Series Sequences is Meaningless (PDF) — Clustering of time series subsequences is meaningless. More concretely, clusters extracted from these time series are forced to obey a certain constraint that is pathologically unlikely to be satisfied by any data set, and because of this, the clusters extracted by any clustering algorithm are essentially random. While this constraint can be intuitively demonstrated with a simple illustration and is simple to prove, it has never appeared in the literature. We can justify calling our claim surprising since it invalidates the contribution of dozens of previously published papers. We will justify our claim with a theorem, illustrative examples, and a comprehensive set of experiments on reimplementations of previous work. From 2003, warning against sliding window techniques.
  3. Toolkits for the Mind (MIT TR) — Programming–language designer Guido van Rossum, who spent seven years at Google and now works at Dropbox, says that once a software company gets to be a certain size, the only way to stave off chaos is to use a language that requires more from the programmer up front. “It feels like it’s slowing you down because you have to say everything three times,” van Rossum says. Amen!
  4. Robots Roam Earth’s Imperiled Oceans (Wired) — It’s six feet long and shaped like an airliner, with two wings and a tail fin, and bears the message, “OCEANOGRAPHIC INSTRUMENT PLEASE DO NOT DISTURB.” All caps considered, though, it’s a more innocuous epigram than the one on a drone I saw back at the dock: “Not a weapon — Science Instrument.”
Comment
Four short links: 10 April 2015

Four short links: 10 April 2015

Graph Algorithm, Touchy Robots, Python Bolt-Ons, and Building Data Products

  1. Exact Maximum Clique for Large or Massive Real Graphs — explanation of how BBMCSP works.
  2. Giving Robots and Prostheses the Human Touchthe team, led by mechanical engineer Veronica J. Santos, is constructing a language of touch that both a computer and a human can understand. The researchers are quantifying this with mechanical touch sensors that interact with objects of various shapes, sizes, and textures. Using an array of instrumentation, Santos’ team is able to translate that interaction into data a computer can understand. The data is used to create a formula or algorithm that gives the computer the ability to identify patterns among the items it has in its library of experiences and something it has never felt before. This research will help the team develop artificial haptic intelligence, which is, essentially, giving robots, as well as prostheses, the “human touch.”
  3. boltons — things in Python that should have been builtins.
  4. Everything We Wish We’d Known About Building Data Products (DJ Patil and RusJan Belkin) — Data is super messy, and data cleanup will always be literally 80% of the work. In other words, data is the problem. […] “If you’re not thinking about how to keep your data clean from the very beginning, you’re fucked. I guarantee it.” […] “Every single company I’ve worked at and talked to has the same problem without a single exception so far — poor data quality, especially tracking data,” he says.“Either there’s incomplete data, missing tracking data, duplicative tracking data.” To solve this problem, you must invest a ton of time and energy monitoring data quality. You need to monitor and alert as carefully as you monitor site SLAs. You need to treat data quality bugs as more than a first priority. Don’t be afraid to fail a deploy if you detect data quality issues.
Comments: 2

Open source won, so what’s next?

What to expect at OSCON 2015.

OSCON 2014 show floor

Twenty years ago, open source was a cause. Ten years ago, it was the underdog. Today, it sits upon the Iron Throne ruling all it surveys. Software engineers now use open source frameworks, languages, and tools in almost all projects.

When I was putting together the program for OSCON with the other program chairs, it occurred to me that by covering “just” open source, we weren’t really leaving out all that much of the software landscape. It seems open source has indeed won, but let’s not gloat; let’s make things even better. Open source has made many great changes to software possible, but the spirit of the founding community goes well beyond code. Read more…

Comment
Four short links: 9 April 2015

Four short links: 9 April 2015

Robot Personalities, Programmer Competency, Docker Dependencies, and Large Files in Git

  1. Google’s Patent on Virtual People Personalities — via IEEE Spectrum, who are not bullish, a method for downloadable personalities. Prior art? Don’t talk to me about prior art. The only thing more depressing than this patent is the tech commentary that fails to cite Hitchhiker’s Guide to the Galaxy.
  2. Programmer Competency Matrix — a rubric for developer development.
  3. Aviator — Clever’s open source service dependency management tool, described here.
  4. Announcing Git’s Large File Storagean improved way to integrate large binary files such as audio samples, data sets, graphics, and videos into your Git workflow..
Comment
Four short links: 7 April 2015

Four short links: 7 April 2015

JavaScript Numeric Methods, Misunderstood Statistics, Web Speed, and Sentiment Analysis

  1. NumericJS — numerical methods in JavaScript.
  2. P Values are not Error Probabilities (PDF) — In particular, we illustrate how this mixing of statistical testing methodologies has resulted in widespread confusion over the interpretation of p values (evidential measures) and α levels (measures of error). We demonstrate that this confusion was a problem between the Fisherian and Neyman–Pearson camps, is not uncommon among statisticians, is prevalent in statistics textbooks, and is well nigh universal in the pages of leading (marketing) journals. This mass confusion, in turn, has rendered applications of classical statistical testing all but meaningless among applied researchers.
  3. Breaking the 1000ms Time to Glass Mobile Barrier (YouTube) —
    See also slides. Stay under 250 ms to feel “fast.” Stay under 1000 ms to keep users’ attention.
  4. Modern Methods for Sentiment AnalysisRecently, Google developed a method called Word2Vec that captures the context of words, while at the same time reducing the size of the data. Gentle introduction, with code.
Comment: 1
Four short links: 1 April 2015

Four short links: 1 April 2015

Tuning Fanout, Moore's Law, 3D Everything, and Social Graph Analysis

  1. Facebook’s Mystery MachineThe goal of this paper is very similar to that of Google Dapper[…]. Both work [to] try to figure out bottlenecks in performance in high fanout large-scale Internet services. Both work us[ing] similar methods, however this work (the mystery machine) tries to accomplish the task relying on less instrumentation than Google Dapper. The novelty of the mystery machine work is that it tries to infer the component call graph implicitly via mining the logs, where as Google Dapper instrumented each call in a meticulous manner and explicitly obtained the entire call graph.
  2. The Multiple Lives of Moore’s LawA shrinking transistor not only allowed more components to be crammed onto an integrated circuit but also made those transistors faster and less power hungry. This single factor has been responsible for much of the staying power of Moore’s Law, and it’s lasted through two very different incarnations. In the early days, a phase I call Moore’s Law 1.0, progress came by “scaling up”—adding more components to a chip. At first, the goal was simply to gobble up the discrete components of existing applications and put them in one reliable and inexpensive package. As a result, chips got bigger and more complex. The microprocessor, which emerged in the early 1970s, exemplifies this phase. But over the last few decades, progress in the semiconductor industry became dominated by Moore’s Law 2.0. This era is all about “scaling down,” driving down the size and cost of transistors even if the number of transistors per chip does not go up.
  3. BoXZY Rapid-Change FabLab: Mill, Laser Engraver, 3D Printer (Kickstarter) — project that promises you the ability to swap out heads to get different behaviour from the “move something in 3 dimensions” infrastructure in the box.
  4. SociaLite (Github) — a distributed query language for graph analysis and data mining. (via Ben Lorica)
Comment: 1
Four short links: 30 March 2015

Four short links: 30 March 2015

Philosophical Research, Reading Turing, Security Exercises, and Golang Madness

  1. The Trolley and the PsychopathNot only does a “utilitarian” response (“just kill the fat guy”) not actually reflect a utilitarian outlook, it may actually be driven by broad antisocial tendencies, such as lowered empathy and a reduced aversion to causing someone harm. Questionably expanding scope of claims in the behavioural philosophy research. (via Ed Yong)
  2. Summary of Computing Machinery and Intelligence (1950) by Alan Turing (Jack Hoy) — still interesting and relevant today. cf Why Aren’t We Reading Turing
  3. Exploit Exercisesa variety of virtual machines, documentation, and challenges that can be used to learn about a variety of computer security issues, such as privilege escalation, vulnerability analysis, exploit development, debugging, reverse engineering, and general cyber security issues.
  4. GopherJS — golang to Javascript compiler so you can experience the ease of typed compiled languages in the security and stability of the browser platform.
Comment
Four short links: 26 March 2015

Four short links: 26 March 2015

GPU Graph Algorithms, Data Sharing, Build Like Google, and Distributed Systems Theory

  1. gunrocka CUDA library for graph primitives that refactors, integrates, and generalizes best-of-class GPU implementations of breadth-first search, connected components, and betweenness centrality into a unified code base useful for future development of high-performance GPU graph primitives. (via Ben Lorica)
  2. How to Share Data with a Statisticiansome instruction on the best way to share data to avoid the most common pitfalls and sources of delay in the transition from data collection to data analysis.
  3. Bazela build tool, i.e. a tool that will run compilers and tests to assemble your software, similar to Make, Ant, Gradle, Buck, Pants, and Maven. Google’s build tool, to be precise.
  4. You Can’t Have Exactly-Once Delivery — not about the worst post office ever. FLP and the Two Generals Problem are not design complexities, they are impossibility results.
Comment
Four short links: 23 March 2015

Four short links: 23 March 2015

Agricultural Robots, Business Model Design, Simulations, and Interoperable JSON

  1. Swarmfarm RoboticsHis previous weed sprayer weighed 21 tonnes, measured 36 metres across its spray unit, guzzled diesel by the bucketload and needed a paid driver who would only work limited hours. Two robots working together on Bendee effortlessly sprayed weeds in a 70ha mung-bean crop last month. Their infra-red beams picked up any small weeds among the crop rows and sent a message to the nozzle to eject a small chemical spray. Bate hopes to soon use microwave or laser technology to kill the weeds. Best of all, the robots do the work without guidance. They work 24 hours a day. They have in-built navigation and obstacle detection, making them robust and able to decide if an area of a paddock should not be traversed. Special swarming technology means the robots can detect each other and know which part of the paddock has already been assessed and sprayed.
  2. Route to Market (Matt Webb) — The route to market is not what makes the product good. […] So the way you design the product to best take it to market is not the same process to make it great for its users.
  3. Explorable Explanations — points to many sweet examples of interactive explorable simulations/explanations.
  4. I-JSON (Tim Bray) — I-JSON is just a note saying that if you construct a chunk of JSON and avoid the interop failures described in RFC 7159, you can call it an “I-JSON Message.” If any known JSON implementation creates an I-JSON message and sends it to any other known JSON implementation, the chance of software surprises is vanishingly small.
Comment
Four short links: 18 March 2015

Four short links: 18 March 2015

Moonshots, Decacorns, Leadership, and Deep Learning

  1. How to Make Moonshots (Astro Teller) — Expecting a person to be a reliable backup for the [self-driving car] system was a fallacy. Once people trust the system, they trust it. Our success was itself a failure. We came quickly to the conclusion that we needed to make it clear to ourselves that the human was not a reliable backup — the car had to always be able to handle the situation. And the best way to make that clear was to design a car with no steering wheel — a car that could drive itself all of the time, from point A to point B, at the push of a button.
  2. Billion-Dollar Math (Bloomberg) — There’s a new buzzword, “decacorn,” for those over $10 billion, which includes Airbnb, Dropbox, Pinterest, Snapchat, and Uber. It’s a made-up word based on a creature that doesn’t exist. “If you wake up in a room full of unicorns, you are dreaming,” Todd Dagres, a founding partner at Spark Capital, recently told Bloomberg News. Not just cute seeing our industry explained to the unwashed, but it’s the first time I’d seen decacorn. (The weather’s just dandy in my cave, thanks for asking).
  3. What Impactful Engineering Leadership Looks Like — aside from the ugliness of “impactful,” notable for good advice. “When engineering management is done right, you’re focusing on three big things,” she says. “You’re directly supporting the people on your team; you’re managing execution and coordination across teams; and you’re stepping back to observe and evolve the broader organization and its processes as it grows.”
  4. cxxnet“a fast, concise, distributed deep learning framework” that scales beyond a single GPU.
Comment