The wisdom of the wisdom of crowds

Fri

Dec 15
2006

listen

The wisdom of the wisdom of crowds

News.com has a very interesting article about the use of prediction markets at Google, Yahoo (mentioning Radar's TechBuzz collaboration), Microsoft, and HP. The Microsoft section reinforces the research done at HP on this years ago, that the use of a prediction market to forecast business results helped expose information that people weren't willing to share directly with their bosses:

The goal: to have 25 members of a development team predict when a Microsoft product would ship (this was an internal product, not one sold externally). The prediction market was set up in August 2004, and the product that "had been in the works for a long time" was scheduled to ship in November 2004.

Each "trader" received $50 in their account to start with, and was told that the more accurate their prediction, the more money they would make. The market opened with an initial price of on-time delivery set to 16 2/3 cents.

"The price of 'before November' dropped to zero right away," Proebsting said. "The price of 'on time' in about two to three minutes dropped to 2.3 cents on the dollar." Translated, that's more than 30-to-1 odds against on-time delivery.

Then the woman who was responsible for scheduling started trying to convince her colleagues who were buying and selling future delivery dates. "She was able to talk (on-time delivery) up to around 3 cents," Proebsting said. "People really enjoyed moving the price...They loved this."

"The next day the director comes into my office and said, 'What have you done?'" Proebsting said. But further investigation showed that the product actually was behind schedule, even though nobody was telling management, and it eventually shipped in February.

The article also talks about similar or related projects at the other companies. The question that occurs to me is, do these systems work better in workplaces where honesty in estimates is depressed? In other words, is there a characteristic or set of characteristics that produce unusually accurate prediction markets simply because they route around poor communication? Or, is poor communication inevitable, and a prediction market always a good idea as a result? (I'm definitely not trying to slam Microsoft or its management in the form of the question -- any team at any company could easily have the same symptoms or results. Instead, I appreciate that they shared the story.)

I'm inclined to think you don't need bad management for good prediction markets. I tend to think that even fantastically well-managed environments can still use a reality check when it comes to estimates and targets. I know that when I ask people I work with about when they think something will be done, they inevitably have to think, what does Marc want to hear? and will he think worse of me if I say something else? I hope that I do well at praising people for saying what is true regardless of what I want to be true, but I can imagine that, for some people or some situations or some questions, an anonymous poll will always trump even the best management relationship. If that's true, then this is one of those cases where money (or "money") does better at reducing questions to their essence -- for which it is often derided. The lesson I'd take is that the most effective prediction market would be the one with the greatest anonymity protection. Don't give me a t-shirt if I trade well; instead, put my payoff in a dead-drop account.

tags: | comments: 7 | Sphere It
submit:

Previous | Next

0 TrackBacks

TrackBack URL for this entry: http://blogs.oreilly.com/cgi-bin/mt/mt-t.cgi/5102

Comments: 7

Bruce Hughes [12.15.06 02:17 PM]

For what it's worth, my experience is that the phenomenon doesn't need a poisonous management climate in which to thrive. "Concealed rework" is a dynamically emergent property of people working in groups, and it is hard to avoid.

www2.umassd.edu/systemdynamics/public/90_percent_syndrome.pdf

Tim van Gelder [12.16.06 02:10 AM]

Seems there's a potential conflict of interest in the Microsoft situation - the product's ship date is partly under the development team's (i.e., teh traders') control. As "management" in a micro software enterprise, I know that I not only need the development team to be frank, but I also need them to cooperate in the sense of aligning their efforts with the "official" schedule, which they have to reconcile with their own sense of what has to be done by when. I don't think I'd try a prediction market of the kind described above in our own enterprise since it would introduce one more factor to complicate people's allocation of efforts, even if only at a subtle and unconscious level. However, a different kind of prediction market - one where the traders are independent of the development effort (though they'd still have to be intimately familiar with it, somehow) - might work.

Marc Hedlund [12.16.06 09:45 AM]

Tim, I've heard that criticism before. I'm not quite sure whether I agree with it or not; it's a valid point, but I think it assumes that the outcome of the prediction market is a higher priority than the outcome of the work. That might be a valid assumption, at least for a period of time, but it also might not. It would depend, I think, on the developers, the incentives built into the market, and the urgency of the work.

I believe in HP's first research project, they had sales managers participating in the market, not salespeople. At a larger organization (like Microsoft), that might help -- the individual contributors (so-called) are doing the work, and the managers are using the market to communicate reality up the chain. I just think you probably lose a lot of truth in that first jump.

tony [12.17.06 02:17 AM]

I run a $3m start-up, and it has always been a bit hit and miss forecasting sales - large, lumpy deals that take 6 months from start to finish mean that quarter forecasts are all over the place.

In August, I set up an Inkling market for my management team to trade on our sales number. I offered a fixed, but quite large $ prize for whoever made the most from trading.

A couple of observations:

This las observation goes to Marc's question. Pumping up a sales force or a development team is actually part of _good_ management, not bad - especially of American teams. But an unfortunate consequence of it is that it produces an endemic optimism. This is the culture of the american dream.

I think the same can happen of development teams. My developpment team is in the UK, and takes great pride in its gritty realism. I have not had to set up a market for development schedules - if anything I have to keep pushing on whether we're not being just too pessimistic about when new versions will come out.

Marc Hedlund [12.17.06 07:20 AM]

Thanks, Tony, that's all very interesting to read.

Rick Davies [12.31.06 09:10 AM]

The wider question of how to best explain the varying performance of prediction markets is an interesting one.

This could be done retospectively, by comparing peformance after the event (e.g. a year's worth of predictions). But it would be even better if the resulting theory could then be successfully used to predict which markets would do best in the future.

Have any prediction markets been used to predict which other prediction markets will do the best over x period of time, in generating successfull predictions? (i.e. a meta-prediction market)

Anthony [01.12.07 11:56 AM]

Good stuff... now how do you set up a prediction market, preferably with free/open-source software?