- Seagate Kinetic Storage — In the words of Geoff Arnold: The physical interconnect to the disk drive is now Ethernet. The interface is a simple key-value object oriented access scheme, implemented using Google Protocol Buffers. It supports key-based CRUD (create, read, update and delete); it also implements third-party transfers (“transfer the objects with keys X, Y and Z to the drive with IP address 22.214.171.124”). Configuration is based on DHCP, and everything can be authenticated and encrypted. The system supports a variety of key schemas to make it easy for various storage services to shard the data across multiple drives.
- Masters of Their Universe (Guardian) — well-written and fascinating story of the creation of the Elite game (one founder of which went on to make the Raspberry Pi). The classic action game of the early 1980s – Defender, Pac Man – was set in a perpetual present tense, a sort of arcade Eden in which there were always enemies to zap or gobble, but nothing ever changed apart from the score. By letting the player tool up with better guns, Bell and Braben were introducing a whole new dimension, the dimension of time.
- Micropolar (github) — A tiny polar charts library made with D3.js.
- Introduction to R (YouTube) — 21 short videos from Google.
Disk Over Ethernet, Inside Elite, Polar Charts, and R Videos
Video Editing, Game Engine, Python Debugger, and P2P VPN
Filmic Photogrammetry, Car APIs, Takedowns, and OpenCV for Processing
- Sifted — 7 minute animation set in a point cloud world, using photogrammetry in film-making. My brilliant cousin Ben wrote the software behind it. See this newspaper article and tv report for more.
- Vehicle Tech Out of Sync with Drivers’ Devices — Ford Motor Co. has its own system. Apple Inc. is working with one set of automakers to design an interface that works better with its iPhone line. Some of the same car companies and others have joined the Car Connectivity Consortium, which is working with the major Android phone brands to develop a different interface. FFS. “… you are changing your phone every other year, and the top-of-mind apps are continuously changing.” That’s why Chevrolet, Mini and some other automakers are starting to offer screens that mirror apps from a smartphone.
- Incentives in Notice and Takedown (PDF) — findings summarised in Blocking and Removing Illegal Child Sexual Content: Analysis from a Technical and Legal Perspective: financial institutions seemed to be relatively successful at removing phishing websites while it took on average 150 times longer to remove child pornography.
- OpenCV for Processing (Github) — OpenCV for Processing is based on the official OpenCV Java bindings. Therefore, in addition to a suite of friendly functions for all the basics, you can also do anything that OpenCV can do. And a book from O’Reilly, and it’ll be CC-licensed. All is win. (via Greg Borenstein)
Microvideos for MIcrohelp, Organic Search, Probabilistic Programming, and Cluster Management
- How to Make Help Microvideos For Your Site (Alex Holovaty) — Instead of one monolithic video, we decided to make dozens of tiny, five-second videos separately demonstrating features.
- How Google is Killing Organic Search — 13% of the real estate is organic results in a search for “auto mechanic”, 7% for “italian restaurant”, 0% if searching on an iPhone where organic results are four page scrolls away. SEO Book did an extensive analysis of just how important the top left of the page, previously occupied by organic results actually is to visitors. That portion of the page is now all Google. (via Alex Dong)
- Church — probabilistic programming language from MIT, with tutorials. (via Edd Dumbill)
- mesos — a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark (a new framework for low-latency interactive and iterative jobs), and other applications. Mesos is open source in the Apache Incubator. (via Ben Lorica)
Velocity 2013 Speaker Series
Be honest, have you ever wanted to play Steve Souders for a day and pull some revealing stats or trends about some web sites of your choice? Or maybe dig around the HTTP archive? You can do that and more by setting up your own HTTP Archive.
httparchive.org is a fantastic tool to track, monitor, and review how the web is built. You can dig into trends around page size, page load time, content delivery network (CDN) usage, distribution of different mimetypes, and many other stats. With the integration of WebPagetest, it’s a great tool for synthetic testing as well.
You can download an HTTP Archive MySQL dump (warning: it’s quite large) and the source code from the download page and dissect a snapshot of the data yourself. Once you’ve set up the database, you can easily query anything you want.
You need MySQL, PHP, and your own webserver running. As I mentioned above, HTTP Archive relies on WebPagetest—if you choose to run your own private instance of WebPagetest, you won’t have to request an API key. I decided to ask Patrick Meenan for an API key with limited query access. That was sufficient for me at the time. If I ever wanted to use more than 200 page loads per day, I would probably want to set up a private instance of WebPagetest.
To find more details on how to set up an HTTP Archive instance yourself and any further advice, please check out my blog post.
Going back to the scenario I described above: the real motivation is that often you don’t want to throw your website(s) in a pile of other websites (e.g. not related to your business) to compare or define trends. Our digital property at the Canadian Broadcasting Corporation’s (CBC) spans over dozens of URLs that have different purposes and audiences. For example, CBC Radio covers most of the Canadian radio landscape, CBC News offers the latest breaking news, CBC Hockey Night in Canada offers great insights on anything related to hockey, and CBC Video is the home for any video available on CBC. It’s valuable for us to not only compare cbc.ca to the top 100K Alexa sites but also to verify stats and data against our own pool of web sites.
In this case, we want to use a set of predefined URLs that we can collect HTTP Archive stats for. Hence a private instance can come in handy—we can run tests every day, or every week, or just every month to gather information about the performance of the sites we’ve selected. From there, it’s easy to not only compare trends from httparchive.org to our own instance as a performance baseline, but also have a great amount of data in our local database to run queries against and to do proper performance monitoring and investigation.
The beautiful thing about having your own instance is that you can be your own master of data visualization: you can now create more charts in addition to the ones that came out of the box with the default HTTP Archive setup. And if you don’t like Google chart tools, you may even want to check out D3.js or Highcharts instead.
The image below shows all mime types used by CBC web properties that are captured in our HTTP archive database, using D3.js bubble charts for visualization.
3D Code, Malbuffering, p2p Hardware, and Crypto Challenges
- Meshlab — open source, portable, and extensible system for the processing and editing of unstructured 3D triangular meshes.
- HTML5 Video on iOS (Steve Souders) — While it’s true that Mobile Safari on iOS doesn’t buffer any video data as a result of the PRELOAD attribute, it does make other video requests that aren’t counted as “buffered” video. The number and size of the requests and responses depends on the video. For larger videos the total amount of data for these behind-the-scenes requests can be significant.
- Space Monkey (Kickstarter) — distributed encrypted peer-to-peer cloud service using custom hardware. Not open source, which would make me nervous that I was buying a botnet client with storage capability. (via BERG London)
- Matasano Crypto Challenges — Counting is not a hard problem. But cryptography is. There are just a few things you can screw up to get the size of a buffer wrong. There are tens, probably hundreds, of obscure little things you can do to take a cryptosystem that should be secure even against an adversary with more CPU cores than there are atoms in the solar system, and make it solveable with a Perl script and 15 seconds. Don’t take our word for it: do the challenges and you’ll see. People “know” this already, but they don’t really know it in their gut, and we think the reason for that is that very few people actually know how to implement the best-known attacks. So, mail us, and we’ll give you a tour of them.
Email Triage, Pulse Detection, Big Building Data, and Raspberryduino Ardpi
- Triage — iPhone app to quickly triage your email in your downtime. See also the backstory. Awesome UI.
- Webcam Pulse Detector — I was wondering how long it would take someone to do the Eulerian video magnification in real code. Now I’m wondering how long it will take the patent-inspired takedown…
- How Microsoft Quietly Built the City of the Future — The team now collects 500 million data transactions every 24 hours, and the smart buildings software presents engineers with prioritized lists of misbehaving equipment. Algorithms can balance out the cost of a fix in terms of money and energy being wasted with other factors such as how much impact fixing it will have on employees who work in that building. Because of that kind of analysis, a lower-cost problem in a research lab with critical operations may rank higher priority-wise than a higher-cost fix that directly affects few. Almost half of the issues the system identifies can be corrected in under a minute, Smith says.
- UDOO (Kickstarter) — mini PC that could run either Android or Linux, with an Arduino-compatible board embedded. Like faster Raspberry Pi but with Arduino Due-compatible I/O.
Video Magnification Code, Copyright MOOC, Open Access Cost-Effectiveness, and SCADA Security (Sucks)
- Eulerian Video Magnification — papers and the MatLab source code for that amazing effect of exaggerating small changes in file. (*This work is patent pending)
- CopyrightX — MOOC on current law of copyright and the ongoing debates concerning how that law should be reformed. Through a combination of pre-recorded lectures, live webcasts, and weekly online seminars, participants in the course will examine and assess the ways in which law seeks to stimulate and regulate creative expression. (via BoingBoing)
- Cost Effectiveness for Open Access Journals — This plot reveals the prestige (Article Influence score) and publication charges for open access journals.
- Results of SANS SCADA Survey 2013 (PDF) — Unfortunately, at this time they seem unable to monitor the PLCs, terminal units and connections to field equipment due to lack of native security in the control systems themselves. (via InfoSecIsland)
Android TV systems provide low-cost interactive care
Video systems can streamline hospital care in all sorts of ways from displaying messages (“Quiet time is 1 to 2 PM today”) to taking patient surveys, showing patients their X-Rays, and helping patients view their records from their beds. But most of these systems lie outside the budgets of small and rural hospitals. Healthcare Information is halving the costs of the systems, largely by deploying Android in their sets, and is selling them to smaller healthcare institutions that could not afford them before. The use of Android also permits hospitals to choose among the hundreds of thousands of standard apps available in App Stores.
Sequence of screens showing path through a patient survey
Video Effects, Old School, Data Set, and Games Numbers
- Atkinson Dithering in Real Time — a Processing app that renders what the video camera sees, as though it were an original Mac black and white image.
- Patching Binaries — a patch for a crashing bug during import of account transactions or when changing a payee of a downloaded transaction in Microsoft Money Sunset Deluxe. Written with no source, simply by debugging the executable as it shipped for XP.
- Book Crossing Dataset — Contains 278,858 users (anonymized but with demographic information) providing 1,149,780 ratings (explicit / implicit) about 271,379 books.
- Network Games Market Update (Cartagena Capital) — The myth that players use mobile only ‘on the go’ has been shattered. Smartphones and tablets are now mainstream gaming platforms in their own right and a significant proportion of players play in stationary use case scenarios. Stats abound, including 38% of tablet gamers play more than five hours per week compared to 20% of mobile phone gamer.