Harnessing Collective Intelligence

I've long made the assertion that one of the central differences between the PC era and the Web 2.0 era is that once the internet becomes platform, rather than just an add-on to the PC, you can build applications that harness network effects, so that they become better the more people use them. I've used the phrase "harnessing collective intelligence" to frame this phenomenon.
 

Yesterday at the Web 2.0 Conference, I hosted a panel on this topic. It featured Jim Buckmaster of Craigslist, Toni Schneider of Automatic (Wordpress), Owen van Natta of Facebook, and Richard Roseblatt, former chairman of Myspace, and now CEO of Demand Media.

One of the threads we focused on in the discussion was the difference between "user generated content," which many people focus on, and a far broader, more thought-provoking understanding of how collective intelligence is put to work. Craig Kaplan of predictwallstreet.com asked a question during the panel, and followed up with an email, which I thought was worth sharing:

As I mentioned in my question to the panelists, I feel there is a big difference between user generated content and collective intelligence.
 

For example, PredictWallStreet.com focuses one million unique monthly visitors on predicting whether a stock will close up or down. With the help of our algorithms the community can outperform the market -- something most analysts can't do. That's not user-generated content, that's a cognitive community exhibiting super intelligent behavior.

Wikipedia exhibits super intelligent behavior when it is more comprehensive and more up to date than encyclopedia Britannica. Britannica has the brand, but Wikipedia has the Brains on Board. And with very minimal software, Wikipedia directs millions of minds to create a new and better kind of encyclopedia. That's not just user-generated content. It's a cognitive community exhibiting super intelligent behavior.

Together we form a super intelligence that is a lot smarter than any one of us alone. As you say, Web 2.0 truly is just the froth before the wave. I believe networks of super intelligent cognitive communities are our future.

While I'm not sure I'd use the phrase "super-intelligent," I agree very much with Richard that there's a lot more to harnessing collective intelligence (the new HCI) than user generated content (UCG). Google's PageRank is HCI, but not UGC (although it is derived from the user-generated content of the WWW itself); Wordpress's akismet anti-spam plugin is HCI but not UGC; CraigsList's user moderation features are HCI applied to the problem of unbridled UGC.
 

Even sites that are very explicitly based on user-generated content, like MySpace, at their best bring in other aspects of HCI. During the panel, I mentioned Kathy Sierra's recent observation about MySpace, courtesy of her daughter: ""myspace keeps doing what everybody really wants, and it happens instantly.... As soon as you think of something, it's in there.... It's always evolving. It changes constantly. There's always something new." Richard responded that at MySpace, "product development is marketing." They test features on real customers in real time, trying to learn what they like by offering it to them, keeping what works, and changing what doesn't.

As you can see from Richard's comment, Web 2.0 concepts like harnessing collective intelligence and lightweight, responsive software development are intimately related. They require new competencies, new development models, and new attitudes towards the application development process.