The wisdom of the wisdom of crowds

News.com has a very interesting article about the use of prediction markets at Google, Yahoo (mentioning Radar’s TechBuzz collaboration), Microsoft, and HP. The Microsoft section reinforces the research done at HP on this years ago, that the use of a prediction market to forecast business results helped expose information that people weren’t willing to share directly with their bosses:

The goal: to have 25 members of a development team predict when a Microsoft product would ship (this was an internal product, not one sold externally). The prediction market was set up in August 2004, and the product that “had been in the works for a long time” was scheduled to ship in November 2004.

Each “trader” received $50 in their account to start with, and was told that the more accurate their prediction, the more money they would make. The market opened with an initial price of on-time delivery set to 16 2/3 cents.

“The price of ‘before November’ dropped to zero right away,” Proebsting said. “The price of ‘on time’ in about two to three minutes dropped to 2.3 cents on the dollar.” Translated, that’s more than 30-to-1 odds against on-time delivery.

Then the woman who was responsible for scheduling started trying to convince her colleagues who were buying and selling future delivery dates. “She was able to talk (on-time delivery) up to around 3 cents,” Proebsting said. “People really enjoyed moving the price…They loved this.”

“The next day the director comes into my office and said, ‘What have you done?'” Proebsting said. But further investigation showed that the product actually was behind schedule, even though nobody was telling management, and it eventually shipped in February.

The article also talks about similar or related projects at the other companies. The question that occurs to me is, do these systems work better in workplaces where honesty in estimates is depressed? In other words, is there a characteristic or set of characteristics that produce unusually accurate prediction markets simply because they route around poor communication? Or, is poor communication inevitable, and a prediction market always a good idea as a result? (I’m definitely not trying to slam Microsoft or its management in the form of the question — any team at any company could easily have the same symptoms or results. Instead, I appreciate that they shared the story.)

I’m inclined to think you don’t need bad management for good prediction markets. I tend to think that even fantastically well-managed environments can still use a reality check when it comes to estimates and targets. I know that when I ask people I work with about when they think something will be done, they inevitably have to think, what does Marc want to hear? and will he think worse of me if I say something else? I hope that I do well at praising people for saying what is true regardless of what I want to be true, but I can imagine that, for some people or some situations or some questions, an anonymous poll will always trump even the best management relationship. If that’s true, then this is one of those cases where money (or “money”) does better at reducing questions to their essence — for which it is often derided. The lesson I’d take is that the most effective prediction market would be the one with the greatest anonymity protection. Don’t give me a t-shirt if I trade well; instead, put my payoff in a dead-drop account.