Jul 7

Tim O'Reilly

Tim O'Reilly

Randomness and Success

Ben Lorica wrote in email: "In case you missed it, Leonard Mlodinow had an interesting article a few days ago in the LA Times. [Entitled Meet Hollywood's Latest Genius, it's a] nice case study of how probability and data analysis can challenge conventional wisdom. Whether or not it changes the status quo remains to be seen. [Key points include:]

  • The Illusion of Control: How much of a studio's success can be attributed to the expertise of studio executives, and how much of it can be explained by randomness? According to the article, the "quants" who have studied this question are not convinced that much can be explained by the skills of the executives.

  • Information Cascade: Once a movie has been been "green-lighted" and made, can sophisticated and expensive marketing overcome the fact that the movie is bad? Marketing helps in the first few weeks, after that "information cascade" takes over."

Ben continued: "Speaking of 'experts', the article reminded me of a conversation a few months ago between Malcolm Gladwell and Bill Simmons. [Subscription required.] Gladwell, it turns out, is not a big fan of Isiah Thomas, the highly-paid, "expert" General Manager of the NY Knicks."

"Gladwell: Here's the real question. If I was GM of the Knicks, would I be doing a better job of managing the team than Thomas? I believe, somewhat immodestly, that the answer is yes. And I say this even though it is abundantly clear that Thomas knows several thousand times more about basketball than I do. I've never picked up a basketball. I couldn't diagram a play to save my life. I would put my level of basketball knowledge, among hard core fans, in the 25th percentile.

So why do I think I would be better? There's a famous experiment done by a wonderful psychologist at Columbia University named Dan Goldstein. He goes to a class of American college students and asks them which city they think is bigger -- San Antonio or San Diego. The students are divided. Then he goes to an equivalent class of German college students and asks the same question. This time the class votes overwhelmingly for San Diego. The right answer? San Diego. So the Germans are smarter, at least on this question, than the American kids. But that's not because they know more about American geography. It's because they know less. They've never heard of San Antonio. But they've heard of San Diego and using only that rule of thumb, they figure San Diego must be bigger. The American students know way more. They know all about San Antonio. They know it's in Texas and that Texas is booming. They know it has a pro basketball team, so it must be a pretty big market. Some of them may have been in San Antonio and taken forever to drive from one side of town to another -- and that, and a thousand other stray facts about Texas and San Antonio, have the effect of muddling their judgment and preventing them from getting the right answer.

I smiled when I read Gladwell's comment, because I've often attributed my success as a technology trend spotter to the fact that I'm watching technology "from the cheap seats," far enough away that I focus on the patterns rather than the details. (My focus on pattern recognition goes way back to my first work as a technical writer, when as a Greek and Latin Classics major, I took on the job of writing Fortran and assembly language manuals for a data acquisition processor. I didn't have the faintest idea at first what the technology was about, and approached them like I would a Greek manuscript, identifying the words I didn't understand, and piecing together a broad understanding of how the parts went together, then improving with multiple passes till I got it right. It's a bit like the way a small snip of a hologram contains all the information in the entire image, but gets clearer the more of the original you have.)

I'm much more in sympathy with the Gladwell interview, about the value of common sense informed by a suitable distance from the subject versus "too much" expertise than I am with the LA Times assertion that chance explains success as well as expertise. Both play a huge role in my experience -- not to mention persistence and effort. As Thomas Jefferson remarked, "I'm a great believer in luck and I find the harder I work, the more I have of it." And as Louis Pasteur remarked, "Fortune favors the prepared mind."

But still, the LA Times article on the statistical nature of success is thought provoking, especially with hit-driven fields of endeavor. From the article:

Arthur De Vany, recently retired professor of economics and a member of the Institute for Mathematical Behavioral Sciences at UC Irvine...likes to illustrate the oddities of the film business by comparing films to breakfast cereal. If breakfast cereals were like films, he says, each time we visited the store we would find a large selection of new cereals, and only a few brands that survived from our last trip. Most of these cereals would languish unnoticed, but crowds would gather at certain parts of the aisle, scooping up the popular brands. And yet, within a few weeks, or at most months, even those popular brands would vanish from the shelves. And so our typical cereal breakfast would consist of a product we had never before tried, and very well might not like, but bought because we heard about it from friends or read of it in the newspaper cereal section.

That's precisely how films behave in the marketplace. If we hear good things, we go and perhaps tell others; if we hear bad things, we stay away. It's that process—the way consumers learn from others about the expected quality of the product—that De Vany found is the key to the odd behavior of the film business today. Economists call it an "information cascade."

"People's behavior is simple," De Vany says, "but in the aggregate it leads to a complex system, a system bordering on chaos."

What I find a bit perplexing is that the article fails to address the fact that the "information cascade" isn't random. Whether it's a hit movie or a much-forwarded video on YouTube, that word of mouth information cascade is driven by the ability of the product to excite people by its quality, its novelty, or some other factor that is not at all random. Just because there is also an element of luck at any point in the information cascade doesn't mean that there is no cause, just that the cause is very difficult, perhaps even impossible, to pinpoint in advance. (For a great popular explanation of phenomena like this, from earthquakes and forest fires to wars and the size of cities, Ubiquity, by John Buchanan, which explores how complex systems reach a "critical state" in which a small event sets off enormous consequences, and how, despite the difficulty of predicting these consequences, they occur with a regularity that is rigorous in its pattern. The math that underlies this world of ours is deeper and more subtle than most people suppose.)

tags:   | comments: 7   | Sphere It

Previous  |  Next

0 TrackBacks

TrackBack URL for this entry:

Comments: 7

  Scott Berkun [07.07.06 12:46 PM]

You can find similiar themes in books on the stock market, particularly Malkiels A Random walk down Wallstreet and Fooled by Randomness by Taleb (an odd, but good read).

The takeaway for me is that randomness is misused - a complex system isn't random, it's chaotic. There are reasons movies fail or suceed but they're very complex and impossible to reliably predict. Just because experts fail doesn't mean novices would do better - it may just mean it's complex.

The Gladwell / Isiah Thomas thing is an awful example, as Isiah Thomas has a horrible reputation as a General Manager, worst in the league. To call him an expert, in the context of this argument is entirely unfair. Ask Gladwell if he thinks he could do better than Joe Dumars (Detriot) or any random half-dozen GMs, I doubt he'd have the same opinion.

The San Diego / San Antonio example is a bit of trivia, don't you think? Ask them how to stage a Candian war against Mexico, make computers easier to program, or solve homelessness in the U.S. (or the oil crisis in the world), problems that involve strategy, knowledge and comfort with complexity, and see if any of these "value of ignorance" or "wisdom of crowds" concepts hold as much water.

I'm not saying Mlodinow and Gladwell don't have good points - they're just both prone to picking easy examples and avoiding obvious counter-points.

Regarding Mlodinow's points on studios: being a management guy, I'm most prone to believing the problems are primarily organizational. Most big orgs are prone to bureacracy and committee style thinking, difusing the value of any great talents they have (or encouraging them to leave or phone it in). If you have too many cooks, it doesn't matter how good the individual cooks are.

Ask any film director, a primary creative force in film creation, what they think of working with studios and some of the basic reasons for crappy films are pretty clear (Directors often feel randomized). Same probably goes for book authors, rock bands and television writers.

  kr8tr [07.07.06 03:17 PM]

Actually, San Antonio *is* larger than San Diego (now). Latest population numbers moved San Antonio into the seventh largest city - bigger than San Diego, and bigger even than Dallas.

  Phil Atio [07.08.06 07:23 AM]

Scott Berkun - You are probably right that the proponents of the wisdom of crowds meme are "prone to picking easy examples and avoiding obvious counter-points." But consider one of the main points of James Surowiecki's book "The Wisdom of Crowds" - there are preconditions (for example, independence of the participants' views) for the phenomenon to work. As a matter of fact, the book cites bureaucratic committee-style thinking as one of the scenarios that distorts the wisdom. So I doubt that even the most ardent fans believe the concept "holds water" all the time or even most of the time.

And I must confess that my experience parallels Tim's - viewing the game at a distance, "from the cheap seats," confers vastly better understanding than being an expert. After all, I started out as a preacher and built a successful "adult entertainment" enterprise based on the same insight that motivated Tim.

  Chris Spurgeon [07.08.06 07:57 AM]

I think Leonard Mlodinow's LA Times article is wonderful, but I do wish he had talked about one of the ways studio execs *can* make a acting as a final check that the finished product is competent. A good producer isn't a boundless ego-maniac (well, OK, they may be that too), they're someone who really *does* know the art and science of film making. They can make sure that the edit makes sense, that the script moves well, that the soundtrack isn't overwhelming, stuff like that. They can catch and correct things that can diminish the flow of a film, things that the director is too close to see.

So, you can't just drop a chimp in there, producers *do* have to have a certain level of skill. Of course, if all studio producers are equally good at this skill, than it doesn't matter.

Tim, I'd be curious about some of your biggest winners and losers. Your spotting Perl as a technology about to hit it big and therefore a perfect area for books is a biggie. Have there been O'Reilly book topics that looked like sure wins but never met expectations?

  Kathy Sierra [07.08.06 02:47 PM]

I could not agree more with the notion that in some scenarios, *less* knowledge is more. If we'd had a clue about computer books (or authoring and publishing in general), we'd never have come up with Head First. There were too many reasons--we later learned--why it absolutely positively would never work.

What Bert and I find interesting, Tim, is that you're still able to see so many things in your areas of expertise through "beginner's eyes", despite your long-term experience. Why, for example, were *you* able to look past The Way Things Work when other publishers we talked to were not?

And can you please bottle whatever that is?

  Tim O'Reilly [07.09.06 08:21 AM]

Kathy -- Thanks for the kind words.

I have a nose for important ideas and wonderful new approaches to old subjects, but that doesn't always translate to success. As the old saying goes, "being too early is the same as being wrong."

Two good publishing examples from my career:

1. Jon Udell's phenomenal book, Practical Internet Groupware, sadly now out of print (but still available on Safari) even though it foretold the entire rise of what we now call "social software" and was a key influence in my formulation of ideas about the internet as platform (what I now call Web 2.0). (See the original press release for how I raved about the book, and Jon Udell, who is still one of my heroes and role models.)

2. Programming Jabber. Jabber has indeed become one of the key transports on the net, and the kind of beyond-IM thinking (again, the net as platform) that Programming Jabber explored, but the book never sold all that well.

Of course, sales are not the only model of influence. Both of these books shaped my thinking, and through me, that of a lot of other people who've never heard of them.

But still, to my point, luck and timing matter a lot. What would have happened if your book hadn't been brought to O'Reilly. You might have ended up writing the emasculated version your previous publisher wanted, and the world of technology publishing might never know what it missed.

But back to your question about beginner's eyes: I love the story about Samuel Johnson from Boswell's Tour of Scotland and the Hebrides. They met an old man who said, "Why I thought nothing of climbing that tree when I was a boy." Johnson, who was even older, kicked off his shoes, climbed the tree, and when he came down, said, "I thought nothing of it now." Staying young is part of it.

May we all stay young in mind, curious and receptive to the sparks of insight and creativity that pass like electricity from one person to another.

  Wai Yip Tung [07.12.06 12:51 PM]

Gladwell make plenty of simplistic examples like this that does not stand scrutinization. Which country is larger? Singapore or Kazakhstan? According to his logic, it must be Singapore since people have hardly heard of Kazakhstan.That must be that way a Singapore journalist think when years ago he wrote something like "How come a small country like Kazakhstan can do this and we couldn't..." Turns out Kazakhstan is a huge central Asian country ranked 9th in the world in size that totally dwarf Singapore.

If you treat Gladwell's stuff as talking points they are often stimulating. If you treat them as rules, as the way Gladwell himself like to portrait his work as, they are just junk science.

Post A Comment:

 (please be patient, comments may take awhile to post)

Type the characters you see in the picture above.