I enjoy exposure to new world views, the feeling of one’s brain being stretched to fit a new frame. For that reason, I enjoyed Daniel Suarez’s talk to the Long Now Foundation, entitled “Daemon: Bot-Mediated Realities”. You can listen to the talk as I did, or read Paul Saffo’s summary.
Suarez sees a world in which bots run everything from airplanes and cars to economies and financial systems, and we have decreasing control over them. The automated systems are good for us, enabling us to do more with fewer people, but Suarez reminds us of the downsides: the specialization of knowledge combined with exploitability of software and the easily-imagined situation of still-running code the workings of which nobody understands. It’s certainly changed the way I look at the things around me: inside every intelligent object I wonder, “who knows how these algorithms work? How long will it live?”. Not in a paranoid tinfoil helmet conspiracy way, just becoming aware of the fragility of the software I took for granted.
It was at this point that Suarez’s talk took a turn for the wishful. His solution to the possible nightmarish future of mankind at the mercy of bots that can’t be repaired or replaced was “let’s recreate the Internet, only with strong crypto and human-vouched IDs, and we’ll only permit bots that a quorum of humans have read and validated the source code to, and …” and I had to ask, “dude, have you ever worked with security people?”. The 9/11 terrorists had government IDs, and it’s easy to imagine malicious code doing so in the future. Reading the source code is time-consuming, therefore expensive, and no panacea–bugs can still exist in code that has been audited. The solution to fragile technology isn’t more fragile technology unless you can failover in a redundant array of inexpensive Earths.
I think the greatest value of his talk was from the long view of software: we’re creating actors that live beyond us, and we (software developers and society, the users of the bots) aren’t planning for succession or failure. Come for the world view, but leave early to avoid the questionable solutions.