Early History of the Web Pipes Concept

A few days ago, I gave credit to Jon Udell for the talk at the Perl Conference that first introduced me to the idea of pipes and filters for the web. One reader, Ed, asked in the comments whether it wasn't Andrew Schulman who gave that talk?

After a bit of digging down memory lane with Jon and Andrew, it's still not clear to me whether it was Jon or Andrew who gave the talk I was thinking of, and I should certainly have given Andrew some credit, because I'm pretty sure it was he who brought Jon's ideas to my attention. They both spoke at the conference, and likely on related topics, though there is only a record of Andrew's talk.

However, to complicate things further, Andrew was talking about ideas that started with Jon and some of the hacks he'd been writing about in his Byte column. And being precise, it wasn't first at the conference that I heard about those ideas. Since I was the one who put together the program for the conference, I invited both Andrew and Jon to speak based on what I already knew about their ideas.

In fact, Andrew reminded me that he had proposed a book on the new web programming model, and dug up some outline files from April of 1997. (The conference was in July, if I recall.) And the outline and notes that he dug up indicate some amount of discussion with both me and Jon by that time.

For those of you who are interested in the details of the history, and just how forward thinking both Jon and Andrew were, read on. (I've already posted their memories in the comment thread on the original post, but thought that it was worth giving them both a little more visibility.)

As a side note, it really is amazing how easily we forget the details of the past, and how important it is for future history for us to keep our notes. It gives real perspective on more distant history when you realize how hard it is to remember the sequence of events, and who influenced whom, for something that happened only ten years ago....

But it's also great how ideas seem obvious in retrospect. (And how ideas that seem obvious to the alpha geeks can take years or decades to become "common sense.") Jon wrote:

But really, the whole web pipeline idea was (I assume) completely obvious to the Roy Fieldings of the world who created HTTP and the toolkits like libwww. So obvious that they didn't bother to articulate it, which is all that I did.

(Jon articulated this idea more fully in a blog post this morning entitled Annotate the web, then rewire it.) And Andrew wrote:

Yes, the "web as an API", "web sites are subroutines", etc. theme all did seem pretty obvious at the time, and today even more so. In fact, the idea of a self-contained PC looks more and more like a detour from the main stream of computing. A necessary one, probably (had to get several hundred million boxes out there to bring down costs?), but a detour nonetheless.

Read on for Jon and Andrew's complete notes.

Jon Udell wrote in email:

The Andrew Schulman talk I remember best -- not sure if it was from that conference or not, but I think it may have been -- was about the magic of URLs as human readable/transferable encapsulations of a whole bunch of computation. I remember him saying, over and over: Look, just 40 bytes, and every Fedex package can have its own home page. I also remember him saying that if MS had designed the URL it would look like a GUID. Which of course, years later, came true in the case of their (ahem, our) blog platform :-)

We should ask him about the pipes theme, I just don't remember. Come to think of it I haven't been in touch with him in forever. Do you know how to reach him?

I'm not exactly sure what my talk was at the first Perl conference. But I did write this in late 1996:

"A powerful capability for ad hoc distributed computing arises naturally from the architecture of the Web. ..." [cleaner copy here]

For me that was the lightbulb moment. So I'd guess the topic of the web as a library of components might have been part of my talk in Aug 1997.

OTOH, now that you mention it, I don't remember you reacting to that theme then, or at the Perl conference the following summer. What you did react to was this:

Calculating Web Mindshare

Which was in 1999.

I remember you saying that this web pipeline stuff should be its own book :-)

But really, the whole web pipeline idea was (I assume) completely obvious to the Roy Fieldings of the world who created HTTP and the toolkits like libwww. So obvious that they didn't bother to articulate it, which is all that I did.

Andrew Schulman wrote in email:

I think Jon gets precedence here, possibly from an earlier talk at the same conference. My reasoning below:

Let's see, looks like 1st Perl Conference was August 1997. I remember both Eric Raymond and I spoke. Here's a report from the time:

Keynote - The Web as an API - Andrew Schulman

Andrew Schulman showed how the Web was becoming the command line of the future.

Andrew talked about the change in the concept of what computers can do because of the web. Specifically how some companies that make operating systems and GUI word processors have said that it would be impossible to be doing things that the web is now doing, so programmers are now having a whole new way of doing things open to them.

A large part of time was spent on how complex URLs can be used to cause programs to run on another machine and produce large ranges of data. (e.g. http://www.zip2.com/scripts/map.dll?type=jdir&sType=street& dType=street&dType=street&dstreet=103+MORRIS+ST&dcity=SEBASTOPOL& dstate=CA&streetaddr=N+1st+St&userid=1184558&userpw=xtv0J_txAwt8t E_FD0C&version=91450&java=no&sType=street&streetaddr=1st+st.& city=san+jose&state=CA&ccity=SEBASTOPOL&cstate=CA&ck=3599047&adrVer= 872139786&ver=d3.0 )

His talk is at: http://www.sonic.net/~undoc/perl/talk/webapi1.html

And what do you know (I sure didn't remember), the talk is still there:

"The Web is the API"

... "Distributed Computation in the Guise of Hypertext!"

... "Snarfing Cycles

Or, using URLs to control processes on some other machine"

Using then-amazing example of tracking UPS packages. And there it is:

"As Jon Udell pointed out in his Perl conference talk, the UPS form was probably the first major example where a company used the web to 'open up its business procedures to its customers.'"

With a link to http://dev5.byte.com/perlcon/perlcon.zip, which is busted. And not at archive.org. But clearly before my talk. Earlier at the same conference?

And later, Andrew wrote:

In addition to my previous email about the 1997 Perl Conference and the reference I made in my talk to Jon's work, I remembered that I had explicitly discussed "pipes" in a book that I started writing on the subject. One thing is that I was almost oblivious to the security implications, but I did identify the problem of sites changes out from under a program ("bit rot", brittleness). It looks like I posted this material to sonic.net in April 1997:

... Using what seem at first like some "stupid web tricks" (hooking up NetCraft to the URL-minder [one of Jon Udell's programs]), this chapter will show how to pipe one CGI process into another. That you can do this suggests that web sites are really tools, or software components. So perhaps a better chapter title would be "The Tools Approach to the Web."...

[And in] http://www.sonic.net/~undoc/book/chap5.html

... the point is how multiple processes on the web can sometimes be combined, in much the same the same that the output from one Unix program can be piped into another. Web Pipes, really. Problem: "bit rot", brittleness....

And yet another reference to Jon, so it really does seem he was way before me on all this:

... CGI processes: I say "processes" rather than "programs" to emphasize that these things are running today, right now, on machines across the planet, and you can employ them, without having to write one yourself. You do have to figure out what their "API" is, though (see Jon Udell articles on implicit CGI APIs). And this API can change out from under you (so what else is new?)....

... This chapter should also discuss CGI processes whose output is an embedable image: Web-Counter, US Naval Observatory time, etc. Not pipe-able, but embeddable...

(NB: All this would be great prior art on various web services patents.)