- Computer Software Archive (Jason Scott) — The Internet Archive is the largest collection of historical software online in the world. Find me someone bigger. Through these terabytes (!) of software, the whole of the software landscape of the last 50 years is settling in. (And documentation and magazines and …). Wow.
- 7 in 10 Doctors Have a Self-Tracking Patient — the most common ways of sharing data with a doctor, according to the physicians, were writing it out by hand or giving the doctor a paper printout. (via Richard MacManus)
- opsmezzo — open-sourced provisioning tools from the Nodejitsu team. (via Nuno Job)
- Hacking Secret Ciphers with Python — teaches complete beginners how to program in the Python programming language. The book features the source code to several ciphers and hacking programs for these ciphers. The programs include the Caesar cipher, transposition cipher, simple substitution cipher, multiplicative & affine ciphers, Vigenere cipher, and hacking programs for each of these ciphers. The final chapters cover the modern RSA cipher and public key cryptography.
Learn to build event-driven client and server applications
I want to build a web server, a mail server, a BitTorrent client, a DNS server, or an IRC bot—clients and servers for a custom protocol in Python. And I want them to be cross-platform, RFC-compliant, testable, and deployable in a standardized fashion. What library should I use?
Twisted is a “batteries included” networking engine for writing, testing, and deploying event-driven clients and servers in Python. It comes with off-the-shelf support for popular networking protocols like HTTP, IMAP, IRC, SMTP, POP3, IMAP, DNS, FTP, and more.
To see just how easy it is to write networking services using Twisted, let’s run and discuss a simple Twisted TCP echo server:
from twisted.internet import protocol, reactor
def dataReceived(self, data):
def buildProtocol(self, addr):
With Twisted installed, if we save this code to echoserver.py and run it with python echoserver.py, clients can now connect to the service on port 8000, send it data, and get back their echoed results. Read more…
Software Archive, Self-Tracking, Provisioning, and Python Ciphers
Establishing an effective organization for large-scale growth
In the open source and free software movement, we always exalt community, and say the people coding and supporting the software are more valuable than the software itself. Few communities have planned and philosophized as much about community-building as ZeroMQ. In the following posting, Pieter Hintjens quotes from his book ZeroMQ, talking about how he designed the community that works on this messaging library.
There are, it has been said (at least by people reading this sentence out loud), two ways to make really large-scale software. Option One is to throw massive amounts of money and problems at empires of smart people, and hope that what emerges is not yet another career killer. If you’re very lucky and are building on lots of experience, have kept your teams solid, and are not aiming for technical brilliance, and are furthermore incredibly lucky, it works.
But gambling with hundreds of millions of others’ money isn’t for everyone. For the rest of us who want to build large-scale software, there’s Option Two, which is open source, and more specifically, free software. If you’re asking how the choice of software license is relevant to the scale of the software you build, that’s the right question.
The brilliant and visionary Eben Moglen once said, roughly, that a free software license is the contract on which a community builds. When I heard this, about ten years ago, the idea came to me—Can we deliberately grow free software communities?
Learn to be a better programmer by taking charge of your interests
If you want to be a better programmer, a good first step would be to choose an area of software development to take additional responsibility for. Now, when we say “responsibility,” we don’t mean the sort of “you’re to blame and you accept it” responsibility that is so commonly thought of. Instead, we mean the willingness to take charge or the willingness to do something about an area.
So select out some area of software development, and decide to be a bit more responsible for it than one was before. The “area” could be simply some additional part of the current project you work on, the whole project itself, some type of software development that you haven’t done before, some aspect of software development you’d like to know more about, etc. If you’re feeling adventurous, try deciding that you’re personally responsible for the quality of the entire software project you’re working on. If you do, you may be surprised at how much easier this makes your life. When you’re trying to maintain the quality of only a piece of a software project, it’s very difficult. You’re surrounded by complexity or confusion, and you have to fend it off at every turn. But when you look at the project as a whole instead and try to decide what should be done with it as a whole, the solution presents itself much more readily. Now, it may seem like quite a bit more work to resolve the problems of the whole project, and it is. But it’s considerably more satisfying, tremendously more effective, and will bring you up to seniority as a software developer much more quickly than just trying to solve the problems of your one particular area.
O'Reilly's new site for all things related to programming.
Welcome to O’Reilly Media’s Programming blog, our resource for all things related to programming. Whether you’re a professional developer, hardcore hacker, or student, I hope this site provides you with interesting ideas, ways to learn new skills, exposure to alpha geeks, and the opportunity to interact with our talented and unique editors. The best part of my days are conversations with our editors and authors, and I’d like you to benefit from the same exchange of ideas.
We’re building this blog to meet your needs with news, information, and analysis. Whether you work on front ends, back ends, or middleware, open source software or commercial, there’s something here for you. Over the coming weeks and months we’ll be publishing how-to information, interviews, and our opinions in addition to exposing you to O’Reilly’s vast products and services.
We’ve seen an explosion of interest in the creation of software over the last two years. Groups like Codecademy promise to teach anyone willing to code, even people like Mayor Bloomberg of New York; one of the most important factors in the legal fight between Google and Oracle was a judge who knew how to code; and groups like Code for America are transforming the civic landscape.
Open source remains one of the pillars of the programming community, but we’re building a very large tent — a tent that includes Windows developers, iOS developers, Oracle developers, and more. We’ll also pay attention to non-technical issues that affect programmers: jobs, developer culture, and occasional tangents into other obsessions (like food).
O’Reilly’s readers created the world we inhabit. Now, we need to bring a new generation into that world and expand it for those already making a difference. While we have a lot of ideas for the Programming blog, we want to hear your thoughts about what you want to see on the site and how you want to participate. Please let us know what you think through the comments or email me directly.
Our new research report outlines our vision for the coming-together of software and big machines.
The big machines that define modern life — cars, airplanes, furnaces, and so forth — have become exquisitely efficient, safe, and responsive over the last century through constant mechanical refinement. But mechanical refinement has its limits, and there are enormous improvements to be wrung out of the way that big machines are operated: an efficient furnace is still wasteful if it heats a building that no one is using; a safe car is still dangerous in the hands of a bad driver.
It is this challenge that the industrial internet promises to address by layering smart software on top of machines. The last few years have seen enormous advances in software and computing that can handle gushing streams of data and build nuanced models of complex systems. These have been used effectively in advertising and web commerce, where data is easy to gather and control is easy to exert, and marketers have rejoiced.
Thanks to widespread sensors, pervasive networks, and standardized interfaces, similar software can interact with the physical world — harvesting data, analyzing it in context, and making adjustments in real-time. The same data-driven approach that gives us dynamic pricing on Amazon and customized recommendations on Foursquare has already started to make wind turbines more efficient and thermostats more responsive. It may soon obviate humans as drivers and help blast furnaces anticipate changes in electricity prices. Read more…
Being both liberal and safe in programming is hard
Recent discoveries of security vulnerabilities in Rails and MongoDB led me to thinking about how people get to write software.
In engineering, you don’t get to build a structure people can walk into without years of study. In software, we often write what the heck we want and go back to clean up the mess later. It works, but the consequences start to get pretty monumental when you consider the network effects of open source.
You might think it’s a consequence of the tools we use—running fast and loose with scripting languages. I’m not convinced. Unusually among computer science courses, my alma mater taught us programming 101 with Ada. Ada is a language that more or less requires a retinal scan before you can use the compiler. It was a royal pain to get Ada to do anything you wanted: the philosophical inverse of Perl or Ruby. We certainly came up the “hard way.”
I’m not sure that the hard way was any better: a language that protects you from yourself doesn’t teach you much about the problems you can create.
But perhaps we are in need of an inversion of philosophy. Where Internet programming is concerned, everyone is quick to quote Postel’s law: “Be conservative in what you do, be liberal in what you accept from others.”
The fact of it is that being liberal in what you accept is really hard. You basically have two options: look carefully for only the information you need, which I think is the spirit of Postel’s law, or implement something powerful that will take care of many use cases. This latter strategy, though seemingly quicker and more future-proof, is what often leads to bugs and security holes, as unintended applications of powerful parsers manifest themselves.
My conclusion is this: use whatever language makes sense, but be systematically paranoid. Be liberal in what you accept, but conservative about what you believe.
Ford's OpenXC platform opens up real-time drivetrain data.
OpenXC (Ford Motor) — Ford has taken a significant step in turning its cars into platforms for innovative developers. OpenXC goes beyond the Ford Developer Program, which opens up audio and navigation features, and lets developers get their hands on drivetrain and auto-body data via the on-board diagnostic port. Once you’ve built the vehicle interface from open-source parts, you can use outside intelligence — code running on an Android device — to analyze vehicle data.
Of course, as outside software gets closer to the drivetrain, security becomes more important. OpenXC is read-only at the moment, and it promises “proper hardware isolation to ensure you can’t ‘brick’ your $20,000 investment in a car.”
Still, there are plenty of sophisticated data-machine tieups that developers could build with read-only access to the drivetrain: think of apps that help drivers get better fuel economy by changing their acceleration or, eventually, apps that optimize battery cycles in electric vehicles.
Drivers with Full Hands Get a Backup: The Car (New York Times) — John Markoff takes a look at automatic driver aides — tools like dynamic cruise control and collision-avoidance warnings that represent something of a middle ground between driverless cars and completely manual vehicles. Some features like these have been around for years, many of them using ultrasonic proximity sensors. But some of these are special, and illustrative of an important element of the industrial Internet: they rely on computer vision like Google’s driverless car. Software is taking over some kinds of machine intelligence that had previously resided in specialized hardware, and it’s creating new kinds of intelligence that hadn’t existed in cars at all. Read more…
Unraveling what programming will need for the next 10 years.
Programming is changing. The PC era is coming to an end, and software developers now work with an explosion of devices, job functions, and problems that need different approaches from the single machine era. In our age of exploding data, the ability to do some kind of programming is increasingly important to every job, and programming is no longer the sole preserve of an engineering priesthood.Over the course of the next few months, I’m looking to chart the ways in which programming is evolving, and the factors that are affecting it. This article captures a few of those forces, and I welcome comment and collaboration on how you think things are changing.
Where am I headed with this line of inquiry? The goal is to be able to describe the essential skills that programmers need for the coming decade, the places they should focus their learning, and differentiating between short term trends and long term shifts. Read more…
If we're going to build useful applications on top of the industrial Internet, we must ensure the components interoperate.
One of the most interesting points made in GE’s “Unleashing the Industrial Internet” event was GE CEO Jeff Immelt’s statement that only 10% of the value of Internet-enabled products is in the connectivity layer; the remaining 90% is in the applications that are built on top of that layer. These applications enable decision support, the optimization of large scale systems (systems “above the level of a single device,” to use Tim O’Reilly’s phrase), and empower consumers.
Given the jet engine that was sitting on stage, it’s worth seeing how far these ideas can be pushed. Optimizing a jet engine is no small deal; Immelt said that the engine gained an extra 5-10% efficiency through software, and that adds up to real money. The next stage is optimizing the entire aircraft; that’s certainly something GE and its business partners are looking into. But we can push even harder: optimize the entire airport (don’t you hate it when you’re stuck on a jet waiting for one of those trucks to push you back from the gate?). Optimize the entire air traffic system across the worldwide network of airports. This is where we’ll find the real gains in productivity and efficiency.
So it’s worth asking about the preconditions for those kinds of gains. It’s not computational power; when you come right down to it, there aren’t that many airports, aren’t that many flights in the air at one time. There are something like 10,000 flights in the air at one time, worldwide; and in these days of big data, and big distributed systems, that’s not a terribly large number. It’s not our ability to write software; there would certainly be some tough problems to solve, but certainly nothing as difficult as, say, searching the entire web and returning results in under a second. Read more…