I was talking the other day with Peter Norvig of Google. He tells an interesting story. Google hires temp workers to evaluate search results. Often when the raters and the system disagree, there’s a bug, which Google fixes by tweaking the algorithms. But sometimes, the system is smarter than the raters. A good example: Glacier Bay. The human raters were convinced that Glacier Bay National Park was the right result. But a closer examination of what John Battelle calls “the database of intentions” showed that the system was indeed right: most users were looking for the Glacier Bay Faucet Company, which none of the raters had ever heard of. Google solved the problem by splitting the result page.
(Interestingly, they also do this kind of treatment for flicker because of flickr. Apparently these split results are available for 2-3% of searches.)