Thomas Lord posted an interesting question in a comment on one of my recent posts: “I have a question about how the “Radar” works. Are you tracking Erlang? or following the broader trend around the pi calculus? Is Erlang interesting to you as a technological idea? Or as a particular product?”. I realized we haven’t talked much about what we actually do here, so I thought today I’d take the time to talk about that.
Dale and Tim have great noses for the future, and were right in the thick of things with the commercialization of Unix, popular uptake of the Internet, Open Source, Peer-to-Peer, and Web 2.0. Their methodology is pretty simple: in hacks, research, and startups by alpha-geeks we can often catch early glimpses of what will later be mainstream products or trends. So when Dale saw Pei Wei working on an X11 viewer for this thing called the “World Wide Web”, he thought “everyone can use this”. The O’Reilly Radar is an attempt to scale this beyond Tim and Dale.
So we see trends like Web 2.0, the growing need for concurrency, ubiquitous machine learning, and the importance of operations. We look to see what alpha geeks are doing in those spaces, find the bits that resonate, hold them up and say “this is what the future holds”. We do this on the Radar blog, in Release 2.0, in research reports, in conferences like Velocity, and in the talks we give.
O’Reilly’s business model is obviously predicated on this kind of future thinking—typical animal books take nine months to hit the shelves and it’s hard to launch a conference with shorter lead time (unless you’re Dave McClure!). You might think we’d keep the best ideas for ourselves and publish the rest, but we don’t. We hope the rest of O’Reilly listens to what we say, but we don’t run departments like conferences or books. As Marc Hedlund is fond of quoting, “Don’t worry about people stealing your ideas. If your ideas are any good, you’ll have to shove them down people’s throats”.
Our process isn’t scientific research, where you come up with a hypothesis and then conduct experiments to disprove that hypothesis. What would an experiment to disprove the hypothesis “concurrency is moving from a niche to the mainstream but it’s still largely an unsolved problem” look like? We try to quantify trends wherever we can, but at its heart this is an attempt to train and employ our instincts.
So Erlang and Haskell are interesting to us because we see alpha-geeks learning and playing with them and they have a “we make parallel code easier” story that fits with the trend we see of people struggling to figure out how to take advantage of multicore systems. We look for datapoints like “Amazon built SimpleDB in Erlang” that would confirm the hypothesis “Erlang can be used in the mainstream”, and we also look for failures that might disprove that hypothesis. Such a failure might be “we build this in Erlang but couldn’t keep a team together to run it, so had it rewritten in C++”. In this mindset, Yahoo! Stores is a failure for Lisp and not a success (sorry, Paul!).
That’s why we don’t try to break every story. We leave that to our friends at ReadWriteWeb and TechCrunch. As Tim’s said on many occasions, we “amplify the faint signals of the alpha-geeks.” It’s fun and we get to meet interesting people and think about the way things should be instead of the way they are. Hope that answers your question, Tom!