- Aaron’s Army — powerful words from Carl Malamud. Aaron was part of an army of citizens that believes democracy only works when the citizenry are informed, when we know about our rights—and our obligations. An army that believes we must make justice and knowledge available to all—not just the well born or those that have grabbed the reigns of power—so that we may govern ourselves more wisely.
- Vaurien the Chaos TCP Monkey — a project at Netflix to enhance the infrastructure tolerance. The Chaos Monkey will randomly shut down some servers or block some network connections, and the system is supposed to survive to these events. It’s a way to verify the high availability and tolerance of the system. (via Pete Warden)
- Foto Forensics — tool which uses image processing algorithms to help you identify doctoring in images. The creator’s deconstruction of Victoria’s Secret catalogue model photos is impressive. (via Nelson Minar)
- All Trials Registered — Ben Goldacre steps up his campaign to ensure trial data is reported and used accurately. I’m astonished that there are people who would withhold data, obfuscate results, or opt out of the system entirely, let alone that those people would vigorously assert that they are, in fact, professional scientists.
Learn to build event-driven client and server applications
I want to build a web server, a mail server, a BitTorrent client, a DNS server, or an IRC bot—clients and servers for a custom protocol in Python. And I want them to be cross-platform, RFC-compliant, testable, and deployable in a standardized fashion. What library should I use?
Twisted is a “batteries included” networking engine for writing, testing, and deploying event-driven clients and servers in Python. It comes with off-the-shelf support for popular networking protocols like HTTP, IMAP, IRC, SMTP, POP3, IMAP, DNS, FTP, and more.
To see just how easy it is to write networking services using Twisted, let’s run and discuss a simple Twisted TCP echo server:
from twisted.internet import protocol, reactor
def dataReceived(self, data):
def buildProtocol(self, addr):
With Twisted installed, if we save this code to echoserver.py and run it with python echoserver.py, clients can now connect to the service on port 8000, send it data, and get back their echoed results. Read more…
DevOps is as much about culture as it is about tools.
Operations professionals live in a wind tunnel. If you can imagine one of those game show glass boxes, where a contestant stands inside, the door shuts, and money blows around in a whirlwind, you’ve got a good idea of what Operations feels like much of the time. While you’re trying to grab one technology, another has forced itself across your eyes demanding attention.
The incredible growth of an industry that didn’t really even exist fifteen years ago has provided us with endless opportunity and innovations. It’s also required us to be on the forefront of many new technologies in a way other professions aren’t. The constant drive towards the next technology, the next platform, and the next idea has stratified our organizations, creating specializations in areas like networking, storage, security, data sciences, and a myriad of other functions that challenge our ability to work with our colleagues as a cohesive team.
Establishing an effective organization for large-scale growth
In the open source and free software movement, we always exalt community, and say the people coding and supporting the software are more valuable than the software itself. Few communities have planned and philosophized as much about community-building as ZeroMQ. In the following posting, Pieter Hintjens quotes from his book ZeroMQ, talking about how he designed the community that works on this messaging library.
There are, it has been said (at least by people reading this sentence out loud), two ways to make really large-scale software. Option One is to throw massive amounts of money and problems at empires of smart people, and hope that what emerges is not yet another career killer. If you’re very lucky and are building on lots of experience, have kept your teams solid, and are not aiming for technical brilliance, and are furthermore incredibly lucky, it works.
But gambling with hundreds of millions of others’ money isn’t for everyone. For the rest of us who want to build large-scale software, there’s Option Two, which is open source, and more specifically, free software. If you’re asking how the choice of software license is relevant to the scale of the software you build, that’s the right question.
The brilliant and visionary Eben Moglen once said, roughly, that a free software license is the contract on which a community builds. When I heard this, about ten years ago, the idea came to me—Can we deliberately grow free software communities?
Informed Citizenry, TCP Chaos Monkey, Photographic Forensics, Medical Trial Data
A letter asking for an introduction meets a meditation on self-reliance.
In 1905 Mark Twain wrestled with the sort of request that many readers here have undoubtedly encountered: a new writer with the most tenuous of connections (her uncle was briefly a neighbor in a Nevada mining town) asks Twain to use his influence to get her manuscript published.
It never hurts to carry an introduction from a well-regarded intermediary, as long as your introducer can actually speak to the quality of your work. I think of Twain’s anguished reply every time I’m asked to recommend someone or something I don’t know — or am tempted to ask the same favor of someone else.
Twain’s message is ultimately optimistic: don’t simply try to accumulate influence. Instead, come up with a good idea and sell it on its merits. The world will listen.
- Museum Datasets (Seb Chan) — collections metadata aren’t generally in good quality (often materials are indexed at the “box level”, ie this item number is a BOX and it contains photos of these things), and aren’t all that useful. The story about the Parisian balcony grille is an excellent reminder that the institution’s collections aren’t a be-all and end-all for researchers.
- Hurricane Electric BGP Toolkit — open source tools for diagnosing network problems. (via Nelson Minar)
- Evernote Smart Notebook by Moleskine — computer vision to straighten up photographed pages of the notebook, and the app recognizes special stickers placed on the book as highlights and selections. Nifty micro-use of augmented reality.
- Tupac Coachella Behind the Technology (CBS) — interesting to me is Dr. Dre and Snoop Dogg were considering taking Shakur with them on tour. Just as Hobbit, Tintin, etc. are CG-ing characters to look normal, is the future of “live” spectacle to be this kind of CG show? Will new acts be competing against the Rolling Stones forever?
- wkhtmltopdf (Google Code) — Simple shell utility to convert html to pdf using the webkit rendering engine, and qt. My first piece of “I wrote this, now you can use it too” open source was an HTML to PS converter (this was 1994 or so) via LaTeX. It’s a useful thing, no really.
- Nicira (Wired) — moving network management into software so the network hardware is as dumb as possible. Interesting continuation of the End-to-End principle, whereby smarts live at the edges of the network and the conduits are dumb.
Mobile Data, Startup Ideas, Sci Foo, and Instapaper
- The Coming Mobile Data Apocalypse (Redmonk) — it is clear that the appetite for mobile bandwidth will grow exponentially over the next twelve to eighteen months. With high volumes of smartphones shipping, more and larger form factors entering the market, and the accelerating build out of streaming services, bandwith consumption is set to spike. Equally apparent is that the carriers are ill provisioned to address this demand, both from a network capacity perspective as well as with their pricing structures.
- Hamster Burial Kit and 998 Other Ideas — For Seth Godin’s Alternative MBA program, this week the nine of us came up with 111 business ideas each. But ideas are only valuable when someone (like you) makes something happen. What follows are our 999 business ideas, free for the taking.
- Sci Foo Short Videos — questions posed to Sci Foo attendees with interesting answers. I liked “What Worries You?”
- Instapaper 3 Released — all the features are ones I’ve wanted, which tells me Marco is listening very closely to his customers. Again I say: Instapaper changes the way I use the web as much as RSS did.
Ethics, Regulation, TCP/IP, and Time Travel
- Ethics and Economics — This paper looks at the evidence that suggests that ethical behaviour is good for the economy.
- FCC to Regulate Broadband — Two FCC officials, who spoke on the condition of anonymity, said FCC Chairman Julius Genachowski will announce Thursday that the commission considers broadband service a hybrid between an information service and a utility and that it has sufficient power to regulate Internet traffic under existing law.
- TCP/IP and IMS Sequence Diagrams — watch SYN, ACK, payload, etc. packets to and fro to understand what really happens each time you fetch mail or surf the web. This is what Velocity-type devops performance folks care about.
- How to Build a Time Machine (Daily Mail) — extremely readable article by Stephen Hawking about the possibilities of time travel.