- VP Trees — a data structure for fast spatial searching. A form of nearest neighbour, useful for melodies (PDF) and image retrieval (PDF) and poetry. (via Reddit)
- iYou — iTunes plugin to show you all the stuff your phone collects about you.
- Bar Camps in Primary Schools — NZ teacher deploys bar camps among students. Great things happen.
- Realtime Wikipedia Edits — fascinating and hypnotic and inspirational and appalling and irrelevant all at once.
"algorithm" entries
Improving Uber’s surge pricing
Should algorithmic pricing be the norm rather than the exception?
Request an invitation to Next:Economy, our event aiming to shed light on the transformation in the nature of work now being driven by algorithms, big data, robotics, and the on-demand economy.
Companies want a bigger share of the pie than their competitors, capital wants a bigger share than labor (and labor wants right back), countries want a bigger share than their rivals, but true wealth comes when we make a bigger pie for everyone. Well run markets are a proven way to do that.
Surge pricing is one of Uber’s most interesting labor innovations. Faced with the problem that they don’t have enough drivers in particular neighborhoods or at particular hours, they use market mechanisms to bring more drivers to those areas. If they need more drivers, they raise the price to consumers until enough drivers are incented by the possibility of higher earnings to fill the demand. Pricing is not set arbitrarily. It is driven algorithmically by pickup time — the goal is to have enough cars on the road that a passenger will get a car within 3–5 minutes. (Lyft’s Prime Time pricing is a similar system.) Uber keeps raising the price until the pickup time falls into the desired range.
This is clearly an imperfect system. In one case, surge pricing gouged customers during a crisis, and even in more prosaic situations like bad weather, the end of a sporting event, or a holiday evening, customers can see enormous price hikes. This uncertainty undercuts the fundamental promise of the app, of cheap, on-demand transportation. If you don’t know how much the ride will cost, can you rely on it?
Simpler workflow tools enable the rapid deployment of models
The importance of data science tools that let organizations easily combine, deploy, and maintain algorithms
Data science often depends on data pipelines, that involve acquiring, transforming, and loading data. (If you’re fortunate most of the data you need is already in usable form.) Data needs to be assembled and wrangled, before it can be visualized and analyzed. Many companies have data engineers (adept at using workflow tools like Azkaban and Oozie), who manage1 pipelines for data scientists and analysts.
A workflow tool for data analysts: Chronos from airbnb
A raw bash scheduler written in Scala, Chronos is flexible, fault-tolerant2, and distributed (it’s built on top of Mesos). What’s most interesting is that it makes the creation and maintenance of complex workflows more accessible: at least within airbnb, it’s heavily used by analysts.
Job orchestration and scheduling tools contain features that data scientists would appreciate. They make it easy for users to express dependencies (start a job upon the completion of another job), and retries (particularly in cloud computing settings, jobs can fail for a variety of reasons). Chronos comes with a web UI designed to let business analysts3 define, execute, and monitor workflows: a zoomable DAG highlights failed jobs and displays stats that can be used to identify bottlenecks. Chronos lets you include asynchronous jobs – a nice feature for data science pipelines that involve long-running calculations. It also lets you easily define repeating jobs over a finite time interval, something that comes in handy for short-lived4 experiments (e.g. A/B tests or multi-armed bandits).
Stacks get hacked: The inevitable rise of data warfare
The cycle of good, bad, and stable has happened at every layer of the stack. It will happen with big data, too.
First, technology is good. Then it gets bad. Then it gets stable.
This has been going on for a long time, likely since the invention of fire, knives, or the printed word. But I want to focus specifically on computing technology. The human race is busy colonizing a second online world and sticking prosthetic brains — today, we call them smartphones — in front of our eyes and ears. And stacks of technology on which they rely are vulnerable.
When we first created automatic phone switches, hackers quickly learned how to blow a Cap’n Crunch whistle to get free calls from pay phones. When consumers got modems, attackers soon figured out how to rapidly redial to get more than their fair share of time on a BBS, or to program scripts that could brute-force their way into others’ accounts. Eventually, we got better passwords and we fixed the pay phones and switches.
We moved up the networking stack, above the physical and link layers. We tasted TCP/IP, and found it good. Millions of us installed Trumpet Winsock on consumer machines. We were idealists rushing onto the wild open web and proclaiming it a new utopia. Then, because of the way the TCP handshake worked, hackers figured out how to DDOS people with things like SYN attacks. Escalation, and router hardening, ensued.
We built HTTP, and SQL, and more. At first, they were open, innocent, and helped us make huge advances in programming. Then attackers found ways to exploit their weaknesses with cross-site scripting and buffer overruns. They hacked armies of machines to do their bidding, flooding target networks and taking sites offline. Technologies like MP3s gave us an explosion in music, new business models, and abundant crowd-sourced audiobooks — even as they leveled a music industry with fresh forms of piracy for which we hadn’t even invented laws. Read more…
Strata Week: Add structured data, lose local flavor?
Wikidata's structure vs. diverse knowledge, and a look at the many factors behind Netflix's recommendations.
A critic says Wikidata could undermine Wikipedia's localized information. Also, Netflix explains why its recommendation engine is much more complicated than most people realize.
AI will eventually drive healthcare, but not anytime soon
A merging of artificial intelligence and healthcare is tougher than many realize.
People will eventually get better care from artificial intelligence, but for now, we should keep the algorithms focused on the data that we know is good and keep the doctors focused on the patients.
Strata Week: Unfortunately for some, Uber's dynamic pricing worked
Dynamic pricing angers some Uber users, Hadoop hits 1.0, a possible set back for open-access research.
Uber's dynamic pricing worked as intended on New Year's Eve, but not everyone is happy about that. Elsewhere, Hadoop reaches the 1.0 milestone and proposed legislation seeks to repeal an open-access research policy.
Strata Week: Unfortunately for some, Uber’s dynamic pricing worked
Dynamic pricing angers some Uber users, Hadoop hits 1.0, a possible set back for open-access research.
Uber's dynamic pricing worked as intended on New Year's Eve, but not everyone is happy about that. Elsewhere, Hadoop reaches the 1.0 milestone and proposed legislation seeks to repeal an open-access research policy.
Four short links: 5 December 2011
Spatial Search, Exposing Your Phone's Perfidity, School Unconference, and Wikipedia Viz
Four short links: 18 November 2011
Quantified Learner, Text Extraction, Backup Flickr, and Multitouch UI Awesomeness
- Learning With Quantified Self — this CS grad student broke Jeopardy records using an app he built himself to quantify and improve his ability to answer Jeopardy questions in different categories. This is an impressive short talk and well worth watching.
- Evaluating Text Extraction Algorithms — The gold standard of both datasets was produced by human annotators. 14 different algorithms were evaluated in terms of precision, recall and F1 score. The results have show that the best opensource solution is the boilerpipe library. (via Hacker News)
- Parallel Flickr — tool for backing up your Flickr account. (Compare to one day of Flickr photos printed out)
- Quneo Multitouch Open Source MIDI and USB Pad (Kickstarter) — interesting to see companies using Kickstarter to seed interest in a product. This one looks a doozie: pads, sliders, rotary sensors, with LEDs underneath and open source drivers and SDK. Looks almost sophisticated enough to drive emacs :-)
Four short links: 14 October 2011
Relativity in Short Words, Set Math, Design Inspiration, and Internet of Things
- Theory of Relativity in Words of Four Letters or Less — this does just what it says, and well too. I like it, as you may too. At the end, you may even know more than you do now.
- Effective Set Reconciliation Without Prior Context (PDF) — paper on using Bloom filters to do set union (deduplication) efficiently. Useful in distributed key-value stores and other big data tools.
- Mental Notes — each card has an insight from psychology research that’s useful with web design. Shuffle the deck, peel off a card, get ideas for improving your site. (via Tom Stafford)
- The Internet of Things To Come (Mike Kuniavsky) — Mike lays out the trends and technologies that will lead to an explosion in Internet of Things products. E.g., This abstraction of knowledge into silicon means that rather than starting from basic principles of electronics, designers can focus on what they’re trying to create, rather than which capacitor to use or how to tell the signal from the noise. He makes it clear that, right now, we have the rich petrie dish in which great networked objects can be cultured.