Oct 15

Tim O'Reilly

Tim O'Reilly

Facebook, the Quant Fund Meltdown, and the Techmeme Leaderboard

There's always a risk of self-fulfilling prophecies in social media. Sites or applications become popular, and then stay popular because they are popular. This may be a key to the unusually high concentration of Facebook applications in the "short head" rather than the "long tail." When a system provides powerful feedback mechanisms for herd behavior, it can actually undermine the "wisdom of crowds" rather than enhancing it. (One of James Surowiecki's key observations in his book of that name was that a diverse collection of independently-acting individuals produce the wisdom of crowds effect. To the extent that those individuals reinforce each other's opinions rather than preserving independent decision making, they tend to undermine that group intelligence.)

But there's an even more insidious corollary: when a group of seemingly independent actors are making decisions based on the same limited pool of information, they become more highly correlated, and thus "stupider."

A dramatic example of this phenomenon was documented by Amir E. Khandani and Andrew Lo in a paper entitled What Happened to the Quants in August 2007? (pdf -- via Paul Kedrosky):

During the week of August 6, 2007, a number of high-profile and highly successful quantitative long/short equity hedge funds experienced unprecedented losses. Based on empirical results from TASS hedge-fund data as well as the simulated performance of a specific long/short equity strategy, we hypothesize that the losses were initiated by the rapid unwinding of one or more sizable quantitative equity market-neutral portfolios. Given the speed and price impact with which this occurred, it was likely the result of a sudden liquidation by a multi-strategy fund or proprietary-trading desk, possibly due to margin calls or a risk reduction. These initial losses then put pressure on a broader set of long/short and long-only equity portfolios, causing further losses on August 9th by triggering stop-loss and de-leveraging policies. A significant rebound of these strategies occurred on August 10th, which is also consistent with the sudden liquidation hypothesis. This hypothesis suggests that the quantitative nature of the losing strategies was incidental, and the main driver of the losses in August 2007 was the firesale liquidation of similar portfolios that happened to be quantitatively constructed.

In plain language: despite their ostensibly reliable quantitative construction as market neutral porfolios, these hedge funds nonetheless suffered a meltdown when one or more weaker players were forced to unwind their highly leveraged position, thus putting in motion a cascade too rapid and deep for even a "market neutral" strategy to offset. The problem: everyone was in more or less the same stocks, with the same offsets, so this strategy, a winner when only a few people were using it, became exposed to much more systemic risk merely because the technique had become so widespread.

On a similar note, in a conversation this morning, I learned about the effort at one major investment firm to diversify the input into their quantitative models. The firm has come to realize that they are over-reliant on the same data as everyone else -- what has become, in Paul Kedrosky's words, "the most over-fished pond in the world." (This quest for new sources of meaningful data for financial markets is one of the drivers behind our upcoming Money:Tech Conference, although I have to confess that my own interest is more in understanding what financial markets have to tell us about the future of Web 2.0 and collective intelligence in general. I did a session on this subject at the last etech, as well as a special issue of the Release 2.0 newsletter.)

So what does this have to do with techmeme? When reviewing the Techmeme leaderboard, and then bouncing from there over to Techmeme itself, I was struck by the fact that the surest way to stay up on the leaderboard is to make sure to comment on stories that are currently appearing on the front page of techmeme! This is a self-reinforcing system, where all of the major tech blogs end up covering the same stories. Yes, someone always breaks the news, but you see this amazing pile-on effect. I'm not sure it's healthy.

In thinking about the future of collective intelligence, we need to make sure that we not only think about systems that lead to convergence of opinion, but also ones that ensure divergence, and fresh inputs. The surest way I know to get this is not to pay attention to the breaking news in your own pond, but to find the next community over, and to create new cross connections. Once the connections are well established, move on. (That's why, for example, at Foo Camp, you won't find just folks from communities like open source or web 2.0 -- two areas O'Reilly has become strongly identified with and had a shaping role for -- but new, and potentially explosive mixtures. What happens when folks from synthetic biology meet hedge fund hackers meet roboticists and makers?)

One of the tensions we struggle with all the time is how much energy to put into following areas we've uncovered that are now well known, and how much to spend on exploring the unknown. But it's a reminder, those of you who are pitching stories to us, that we're unlikely to follow up on press releases that are aimed at everyone covering Web 2.0, and far more interested in hearing from people who are living in a slice of the future that hasn't yet become "evenly distributed." After all, that's the key corollary to the William Gibson line that I quote so often ("The future is here. It's just not evenly distributed yet."): once the future does become evenly distributed, it's not the future any more. It's the present.

tags: web 2.0  | comments: 35   | Sphere It

Previous  |  Next

0 TrackBacks

TrackBack URL for this entry:

Comments: 35

  Alexander van Elsas [10.15.07 06:26 AM]

Hi Tim, I like your cross over suggestion in order to mix up information pools to see what fresh ideas come out of that. The "scoop" is still a major driving force in any leaderboard, just look at the number of different blogs that all the unique scoop of Google taking over Jaiku. (but to the reader very recognisable scoop after seing the 20-th headline about it). I tend to skip past the scoop posts quickly and then dive more into the analyzing posts after that because there the author usually throw in his own experiences. Makes better reading usually.
I wrote earlier about the effects of a blogosphere being dominated by a bunch of large sites:

I think there are two issues that rise:
1. New blogs, no matter how hard they try will not succeed in becoming read by others, as their importance is diminished by the bigger ones.
2. Everyone links to the same sites, bringing the same “scoops” making the blogosphere more and more predictable and immune to new or creative thoughts (same blah blah right?)
Crossing over is a good way of getting out of that trap (but who will notice if the crossover takes place on sites with less traffic than the big blogs?).
But I would ad another one as a quick win. True interaction also leads to new distribution of knowledge. I am not so interested in how many people read a blog post I write, but I do like it when they take the time to comment or reply. This "conversation" that starts then often leads to new insights, better ideas, things I didn't know. Smaller blog sites can probably do this a bit easier than the larger ones (due to sheer volume and depth). So let us not all read TechCrunch, TechMeme etc, but also take the time to follow comments or links to much smaller or unknown sites. Who knows what that will bring us!

  Michael R. Bernstein [10.15.07 06:32 AM]

Robert Silverberg wrote an essay in 1993 titled "To much of the rest of the world, America is Science Fiction" that captured another aspect of the 'uneven distribution' that Gibson noted. I wish I could find a copy of the article to link to.

  Remco Kouwenhoven [10.15.07 06:36 AM]

Hi Tim, when you said "The surest way I know to get this is not to pay attention to the breaking news in your own pond, but to find the next community over, and to create new cross connections", Schumpeters famous principle of 'Neue Kombinationen'(new combinations) sprang to mind. His theory of 'creative destruction' was partly based on this principle. Schumpeter's theory of New Combinations forms the basis of many theories on innovation and entrepreneurship.

  Robert Scoble [10.15.07 07:48 AM]

Tim, great point. This is why I read 880 feeds every day in Google Reader. I share my favorite items on my link blog here:

You should check it out. It's quite different than TechMeme because every item on that blog was hand chosen by me. This morning, for instance, I read through 958 items to find about 30 items to share. Much more diversity there.

  Bob Warfield [10.15.07 08:10 AM]

You're describing evolution in ecosystems here, a topic I've written about several times. We see it over an over on the Internet. A reduction of friction makes it faster and easier for a narrow range of "organisms" to dominate. The dangers come when too many focus on the same niches. It gives rise to bubbles where only a very few can win.

I describe this in more detail together with some tactics on my blog:

  Jim Gaynor [10.15.07 09:02 AM]

With fragmented and narrowly focused media in almost every realm, we all run the very real danger of finding ourselves in an echo chamber - where everyone is saying the same thing (because we're just refiltering the same conversations).

David Brin's Earth (wonderfully predictive of today's Internet culture, yet written before gopher and http) mentions the same issue - the filters and fetchers that a character uses to make sense of information overload puts her at risk of a self-filtered worldview. Her solution is to find a hacker to program a sense of intelligent "randomness" into her filters - and intentionally bring her things to challenge her "safe" view.

(And of course, I find it deliciously ironic that Scoble responds with his unique brand of self-promotion.)

  Tim O'Reilly [10.15.07 09:26 AM]

Scoble --

I agree that you do a great service by digesting the feeds you do, but I wouldn't say that the universe you cover is exactly diverse. Take a look at today's feed. 9 of the first ten post are about the web, and the one that isn't is relevant to the same audience. You're definitely in the same pond as techcrunch, techmeme, gigaom, and the other web bloggers, with a bit of outreach into software development.

Compare that with, say, Nascent, the collection of blogs from Nature, the UK science publisher. You could lose yourself in that world for a long time, learning something new on every post.

(Or for that matter, consider Science News. I remember back in the early days of the web thinking that they were the ideal publication to take online -- a print version what we now call a blog, a digest of lots of interesting news across a range of disciplines.)

Or take a look at the TED talks (Video from the conference), or PopTech, which will be streamed live next week.

For that matter, consider the range of coverage of a newspaper like the Christian Science Monitor or the New York Times.

  Viktor [10.15.07 10:32 AM]

Hi Tim,

this is a pretty well knows phenomenon in economics, that in auction theory is called the winner's curse. The idea is very simple, but still you can not escape from it by simply being aware of it. (The best strategy is to loose, but then the best thing is to stay away.)

The theory is based on asymmetric (or simply impartial) information. Because of the lack of information the value you give to a website/article depends on what you see others are giving to it. This way the price naturally goes above its true value, and finally the one who remains in competition has to pay more than the assets really worth.

  Tim O'Reilly [10.15.07 10:44 AM]

Viktor --

Bill Janeway, with whom I've been doing a lot of my web 2.0 meets Wall Street brainstorming, shared a related comment in email to me:

"One fundamental property of financial markets is that liquidity is a function of disagreement. In this case, liquidity - a contested and multi-dimensional term - means "the ability to buy or sell with relatively minimal impact on price". If all owners are sellers or if all non-owners are buyers, liquidity disappears. Again, the first generation of modern finance theorists and practitioners modeled liquidity as a statistically observable attribute of a particular asset (IBM stock, Treasury bonds), rather than as the problematic attribute of a market that has this perverse character: the more you need it, the less there is.

This is what the world has been re-learning over the past several months."

  Tim O'Reilly [10.15.07 10:46 AM]

I realized I should put in another paragraph from Bill Janeway's email to me, as it has many useful links:

"As we have discussed, what one might call the first generation of "modern" financial economists constrained their models to require homogeous, rational agents, all sharing access both to the same data and to the same (true!) model of how the world works and, therefore, the same translation function of data into actionable information. Unfortunately, while the models were tractable, market reality refused to conform with them in a variety of observable ways. More recently, a range of economists have recognized the necessity of accounting for heterogeneous actors with competing views of the world. These include Mordecai Kurz atStanford, Jeremy Stein at Harvard and Juse Scheinkman , Wei Xiong and Harrison Hong all of Princeton."

  BillyWarhol [10.15.07 11:03 AM]

Yeah where do U find the Time to do the NY Times Sunday Crossword Puzzle anymore? ;)) + Scoble finds time to Twitter!!

I guess there is a Huge # of People on the Planet to voraciously consume all this Internet News + Info but for 1 person it's impossible to Keep Up*

Help Me I'm Drowning!!



  Gabe [10.15.07 11:10 AM]

I think there is a self-reinforcing effect with Techmeme, but I believe it's (1.) overstated and (2.) as often good as bad.

On point (1.): glacing at Techmeme presently, there's really a huge diversity of news. Pile-ons, while not infrequent, are not typical either.

On point (2.) pile-ons with a high parroting quotient occur, but Techmeme just as often fosters reactive posts that productively delve deeper into issues.

Regarding the "surest way to stay up on the leaderboard is to make sure to comment on stories that are currently appearing on the front page of techmeme". How many in the Leaderboard seem to follow that strategy? Three? Or, to be generous, six? Out of 100, that's not a lot.

  Deepak [10.15.07 11:49 AM]

It's interesting that even in the science world, you see some of the same effects. People tend to gravitate towards the same stories, although not quite the same way as in the web/tech world (Science Blogs for example tends to drown out a lot of the other interesting discussion, even that from Nascent which has far more variety). This tends to bury some of the more interesting stories resulting in an inward looking community, which I cannot believe does itself any favors.

Technology Review and TED are two examples of resources that do not limit their vision and provide a much better mix of subjects.

  Tim O'Reilly [10.15.07 12:59 PM]

Hey Gabe,

I didn't mean this as an attack on Techmeme per se (though I do think that the introduction of the leaderboard will indeed reinforce the convergence effect. I don't think it's generous to say that six sites are following that strategy today, and I think it's going to get worse. It's just like SEO. It starts out "white hat," but soon people realize that they get results if they game the system.)

I love what you've done with techmeme, but I did want to use it as an illustration of how information markets evolve. Heck, we saw it in our original publishing business. When we started, no one was interested in publishing on the topics we covered. Once we became really successful, everyone was publishing on them, and we always have to break new ground (e.g. Make magazine) and explore new directions to get away from what is now a consensus.

My point is a general one, as I say, not a knock on what you've accomplished. But I do suspect that the leaderboard, while a great PR move on your part, is a bad idea for the ecosystem you cover, and will eventually degrade the value of your service.

  Mike Rogers [10.15.07 01:10 PM]


There was a recent article in the NYT about the cascade effect in science:
that is very relevant: a real-world meme effect with profound consequences across society.

  Andrew Goodman [10.15.07 02:29 PM]

I think Tim's comment is self-evidently valid. If the Techmeme Pile-On becomes a valid, known, technique to generate traffic, then easily-impressed folks who really don't get flattered that often will go hard after that technique. If it works, then it's definitely not healthy. But it probably won't work, because Gabe would then do something to guard against it.

Clearly there are some interesting debates here about the limits to automation, and the benefits of (and limits to) editorial voice.

Is someone willing to comment on how far off a product like Google News is from Techmeme in its general functioning, either a page of news search results, or the home page?

  JP [10.15.07 02:32 PM]

This is a good topic, and for another perspective on it I'm going to throw in a concept from my research.

I do work in swarm based algorithms for routing in mesh networks. That being said, the biggest thing for any swarm based (stigmergy) system is that the system does not converge on one path or solution and stay there, like some techniques do, it has the ability to "back out" of a local optima as routing in mesh networks can be highly chaotic. The way it achieves this is that all edges or routes in its "table" are updated and decayed at the same time (updated when stimulus mandates it, but a function timely degrades it over time, therefore good routes that have recent good throughput stick around, bad ones go away.)

Now, lets that to the business world. For a good "meme aggregator" to function, it needs a decay function, which I think a lot of these systems are missing. These systems that are over-indulging in the "exploit" phase of the market basically set themselves up for massive failure when volitility flares up in the market (and it will! like when SIOC and FOAF sites break down Facebook's 10 billion dollar valuation, and facebook becomes AOL, an old walled garden with some nice data.) Just leave a few "cycles" for "explore", and a market technique might have a little bit more "robust aspect to it.

  Gabe [10.15.07 04:09 PM]

Tim, I was just responding. No slight perceived here.

I just disagree about the "bad for the ecosystem" bit. Techmeme placement remains a bigger incentive than Techmeme Leaderboard placement. And in any case, Techmeme's already in the business of extracting news from the chorus of hype and echoes, which BTW, preceded Techmeme as this pre-launch snapshot shows:

In any case, we'll know soon enough where Techmeme and its "ecosystem" is headed. Just be aware that Techmeme isn't sitting still, and many of the excesses that seems to afflict it are in fact by design.

  Tim O'Reilly [10.15.07 05:00 PM]

Love the ideas from swarm routing, JP. And Andrew, you're totally right that Gabe will be able to add in counter-measures.

  Tim O'Reilly [10.15.07 05:22 PM]

I hear you Gabe. I probably shouldn't have brought you in to the discussion. It really wasn't about techmeme so much as it was the tendency of a bunch of the leading tech blogs to follow each other, creating a perhaps unintentional but still effective keiretsu.

Techmeme does throw up a lot of interesting news. But I do see clusters of "followers," and ran a few experiments to see if piling on got me onto the front page, and fairly reliably, it did. Knowing that many blogs are working this kind of thing hard, I thought I'd call it out.

  Benedict [10.15.07 08:57 PM]

Tim, I think you're dead on. There's a self fulfilling parochialism to all this. The web isn't just about Social Networking, impressions, or The Valley. This is thinking in a bubble that's creating a bubble. A sea change is due.

  Scott Lawton (Blogcosm) [10.16.07 02:32 AM]

Tim: I assume "piling on" just got you one of many little links (?). Surely the clickthru rate on those is much smaller than a cluster's "anchor" posts.

  Adrian [10.16.07 03:08 AM]

One of the most thought-provoking blog articles I've read in many months. The comments too add to the discussion. I picked up this story from news.YC. Interesting how one can analyze the sytem while still being in the system.

  Andrew Kaplan [10.16.07 03:21 AM]

Very interesting story. Being from outside the Silicon Valley/SF area. I believe that it is easy for bloggers, developers, and executives there to lose perspective about the adoption rates and importance of social networks.

Currently outside of the major technology hubs and/or cultural hubs, facebook adoption is slower at best sporadic. The question is whether this means facebook growth projections are overvalued or whether facebook will succumb to the issue of the "masses" pushing out the early adopters/leaders who found each other and connected for idea generation.

I know personally, I subscribe, read and rely on those posts from many of you who are closer to the center of the tech world. I do however build relationships with leaders and developers in this space and experiment with other platforms and therefore can hopefully stay relevant.

Andrew Kaplan

  Mathew Ingram [10.16.07 05:58 AM]

Excellent points as always, Tim -- although I know that deep down, you're enraged by the fact that I'm several steps above you on the Techmeme leaderboard, and this post is a clever strategy to improve your ranking :-)

Just one thing -- in your comment above where you link to Science News (a fantastic publication), you've got the link as, but it's actually


  Chris Rijnders [10.16.07 06:02 AM]

If you read about the theories analyzing how consciousness develops in our brain, the arguments look very similar.

From Gerald Edelman's "Consciousness - How matter becomes imagination": (herein imagine that a neural firing is a good idea written down and neural states are general opinions formed by these ideas and consciousness is the innovation that leads to progress)

"While the patterns of neural firing are remarkably diverse and the repetoire of available neural states is large during waking, the repetoire of available neural states is greatly reduced during slow-wave sleep. Corresponding to this dramatic reduction in the number of differentiated neural states, consciousness is diminished or lost, just as it is in generalized epileptic discharges. Thus, it appears that consciousness requires not just neural activity, but neural activity that changes continually and thus is spatially and temporally differentiated".

So I agree that a "crowds effect" will undermine group intelligence, making us figuratively go to sleep.

  Deva Hazarika [10.16.07 06:58 AM]


I'd contend that what you're calling "well known" here is in reality an illusion created by the phenomenon you describe.

And I think actions like yours reinforce it. Now, there's of course nothing wrong with focusing on real new innovation. But by accepting what is covered as part of this Bay Area web 2.0 tech groupthink as "well known" you foster an attitude shared by many here that results in a lot of point solutions that do something cool, fun, or interesting but don't really provide a compelling enough solution or experience to really have widespread impact on real users who aren't new tech junkies.

  Bediako George [10.16.07 08:46 AM]

This is a problem with Google and the Google algorithm. Google's algorithm relies on the wisdom of crowds to build its initial index. However, since Google is so popular, the crowd now relies on Google.

This suggests to me that over time Google should not be as effective in delivering the "best content" for "established categories" of thought. I simply mean that new web sites, that may be better information sources for a particular type of content, may never be found by crowds because older established web sites have dominated the index, AND the crowds use Google to search. Since they are never found byt the crowd, the crowd does not link to them. Consequently, Google never places them high on the index, hence they never overcome the older established Google indexed content. The cycle continues.

Gaming the Google algorithm can help break this mold. It is actually advantageous to the crowd to allow some amount of gaming as it forces the crowd to indirectly rank new content. It does not give Google an immediate benefit, however, so they do not allow it.

  Krish [10.16.07 04:43 PM]


That NYT article is a baloney. I would any day believe the work done with the public funding than the one with private funding. If you read the article carefully, it calls those people who did science with private funding from food industry as scientists and the scientists from NIH were denoted with quotes to cast suspicion. Being from science background, I will not take the analysis of economists about scientists without a bag full of salt. My comment has nothing to do with what Tim has written. I wanted to point this out because Mike quoted the article here. I am sorry but the article in insanely immature even though the cascade effect may apply to the discussion here.

Now on to the topic, the tech blogosphere is full of people with big egos. The reason for the "me too" pile-ups is because of the fact that the egoistical bloggers want to have their mouth in every pie out there related to tech industry. Sites like Techmeme, may be not intentionally, are contributing for this pile-up. If you take the so called A-list of the tech blogosphere, it is acting exactly like the traditional media these guys were suppose to replace. The money involved and the unlimited egos of the A-Listers are pushing the trend more and more in the pile-up direction. As I said before, techmeme may not be directly responsible for the pile-ups. They are inadvertently helping it happen. Also, by establishing a kinda hierarchy through the leadership board, techmeme is further accelerating this trend. As Tim says, we need to avoid the me too mentality and think outside the bun. In fact, there is a similar historical trend in science. Once upon a time, physicists hanged around only with physicists, mathematicians hanged around only with fellow mathematicians and some elite theoretical physicists and biologists with other biologists. There was growth in science but it was not like how it is in the recent times. Now the overlap is happening and we are seeing great advances in biophysics, bioinformatics, neuroscience, etc. This is the kinda stuff Tim is talking about and it is what is needed for real innovation and also to avoid echo chambers. In my opinion, techmeme is just a camera that takes the snapshots of tech echo chamber and offers it to public in a single page.

  Robert Scoble [10.16.07 10:32 PM]

Ahh, OK, Tim, I see you want some diversity outside of the tech world.

I totally agree. I've been working to interview people outside of my world. I just interviewed a Stanford Pediatrician, for instance and I learned a LOT that I wouldn't have learned just by reading TechMeme.

On the other hand that video wasn't viewed as often as many of the others I've done inside the business.

As to reading feeds. I read 880 feeds and actually look for things that aren't on TechMeme (and I read a pretty damn diverse set of feeds -- just added the two you told me about).

But, I do try to focus down all that stuff to what someone interested in tech would care about.

After all, on the business pages of the New York Times you don't see science stuff all that often either. Do we complain that there isn't enough diversity THERE? I don't think so.

TechMeme is far more diverse than what came before.

Is it diverse enough? Maybe not, but that's what we have RSS aggregators for!

I subscribe to hundreds of smart people in the tech industry (including you).

If my feed doesn't have enough diversity all you gotta do is look in the mirror!

(I linked several of your blog posts lately too. Why? Not because they were on TechMeme but simply because they were more interesting to me than the writing you've been doing on your blog lately.)

  Tim O'Reilly [10.16.07 10:46 PM]

Scoble, I'm not just talking about things that are diverse from tech but of interest to the audience, like science, but even what's next in tech. Take the areas we're covering with Make. It's hacker stuff today, but it will have real business impact soon. There's so much mindblowing stuff that will be all over techmeme or its equivalent a year or two from now.

FWIW, I still find slashdot the most diverse source of interesting tech industry news, with lots of things that surprise and delight me.

  Krish [10.17.07 12:34 PM]


Amen to your Slashdot mention. Even today, it is the best aggregator.

  flaleblanc [10.18.07 06:19 PM]

The dangers of group think have been taught in leadership courses for decades. Thanks for reinforcing the notion that 'everything old is new again.'

  Derek Robinson [10.19.07 12:23 PM]

Herbert Simon was writing, way back in the later 50s early 60s, about the intrinsic self-dooming quality of Wall Street's self-fulfilling prophecies, or self-eating shark-frenzies, as it gets better at predicting the (we're assuming) one true reality that all the actors' predictive devices are trying to model. Such that the better people's predictors get, the worse it gets for everyone. Heh.

  Sean Murphy [10.26.07 09:29 AM]

Tim, as Matthew Ingram mentioned above, the URL for Science News is

I agree that it's a great publication and a good model for what a good magazine website. You should fix the link.

Post A Comment:

 (please be patient, comments may take awhile to post)

Type the characters you see in the picture above.