There’s always a risk of self-fulfilling prophecies in social media. Sites or applications become popular, and then stay popular because they are popular. This may be a key to the unusually high concentration of Facebook applications in the “short head” rather than the “long tail.” When a system provides powerful feedback mechanisms for herd behavior, it can actually undermine the “wisdom of crowds” rather than enhancing it. (One of James Surowiecki’s key observations in his book of that name was that a diverse collection of independently-acting individuals produce the wisdom of crowds effect. To the extent that those individuals reinforce each other’s opinions rather than preserving independent decision making, they tend to undermine that group intelligence.)
But there’s an even more insidious corollary: when a group of seemingly independent actors are making decisions based on the same limited pool of information, they become more highly correlated, and thus “stupider.”
A dramatic example of this phenomenon was documented by Amir E. Khandani and Andrew Lo in a paper entitled What Happened to the Quants in August 2007? (pdf — via Paul Kedrosky):
During the week of August 6, 2007, a number of high-profile and highly successful quantitative long/short equity hedge funds experienced unprecedented losses. Based on empirical results from TASS hedge-fund data as well as the simulated performance of a specific
long/short equity strategy, we hypothesize that the losses were initiated by the rapid unwinding of one or more sizable quantitative equity market-neutral portfolios. Given the
speed and price impact with which this occurred, it was likely the result of a sudden liquidation by a multi-strategy fund or proprietary-trading desk, possibly due to margin calls or
a risk reduction. These initial losses then put pressure on a broader set of long/short and
long-only equity portfolios, causing further losses on August 9th by triggering stop-loss and
de-leveraging policies. A significant rebound of these strategies occurred on August 10th,
which is also consistent with the sudden liquidation hypothesis. This hypothesis suggests
that the quantitative nature of the losing strategies was incidental, and the main driver of
the losses in August 2007 was the firesale liquidation of similar portfolios that happened to
be quantitatively constructed.
In plain language: despite their ostensibly reliable quantitative construction as market neutral porfolios, these hedge funds nonetheless suffered a meltdown when one or more weaker players were forced to unwind their highly leveraged position, thus putting in motion a cascade too rapid and deep for even a “market neutral” strategy to offset. The problem: everyone was in more or less the same stocks, with the same offsets, so this strategy, a winner when only a few people were using it, became exposed to much more systemic risk merely because the technique had become so widespread.
On a similar note, in a conversation this morning, I learned about the effort at one major investment firm to diversify the input into their quantitative models. The firm has come to realize that they are over-reliant on the same data as everyone else — what has become, in Paul Kedrosky’s words, “the most over-fished pond in the world.” (This quest for new sources of meaningful data for financial markets is one of the drivers behind our upcoming <a href=http://conferences.oreilly.com/money2008Money:Tech Conference, although I have to confess that my own interest is more in understanding what financial markets have to tell us about the future of Web 2.0 and collective intelligence in general. I did a session on this subject at the last etech, as well as a special issue of the Release 2.0 newsletter.)
So what does this have to do with techmeme? When reviewing the Techmeme leaderboard, and then bouncing from there over to Techmeme itself, I was struck by the fact that the surest way to stay up on the leaderboard is to make sure to comment on stories that are currently appearing on the front page of techmeme! This is a self-reinforcing system, where all of the major tech blogs end up covering the same stories. Yes, someone always breaks the news, but you see this amazing pile-on effect. I’m not sure it’s healthy.
In thinking about the future of collective intelligence, we need to make sure that we not only think about systems that lead to convergence of opinion, but also ones that ensure divergence, and fresh inputs. The surest way I know to get this is not to pay attention to the breaking news in your own pond, but to find the next community over, and to create new cross connections. Once the connections are well established, move on. (That’s why, for example, at Foo Camp, you won’t find just folks from communities like open source or web 2.0 — two areas O’Reilly has become strongly identified with and had a shaping role for — but new, and potentially explosive mixtures. What happens when folks from synthetic biology meet hedge fund hackers meet roboticists and makers?)
One of the tensions we struggle with all the time is how much energy to put into following areas we’ve uncovered that are now well known, and how much to spend on exploring the unknown. But it’s a reminder, those of you who are pitching stories to us, that we’re unlikely to follow up on press releases that are aimed at everyone covering Web 2.0, and far more interested in hearing from people who are living in a slice of the future that hasn’t yet become “evenly distributed.” After all, that’s the key corollary to the William Gibson line that I quote so often (“The future is here. It’s just not evenly distributed yet.”): once the future does become evenly distributed, it’s not the future any more. It’s the present.