• Print

Network Effects in Data

Nick Carr’s difficulty in understanding my argument that cloud computing is likely to end up a low-margin business unless companies find some way to harness the network effects that are the heart of Web 2.0 made me realize that I use the term “network effects” somewhat differently, and not in the simplistic way many people understand it.

Here’s Nick:

Let’s stop here, and take a look at the big kahuna on the Net, Google, which O’Reilly lists as the first example of a business that has grown to dominance thanks to the network effect. Is the network effect really the main engine fueling Google’s dominance of the search market? I would argue that it certainly is not….

The intelligence embedded in a link is equally valuable to Google whether the person who wrote the link is a Google user or not. In his new post, in other words, O’Reilly is confusing “harnessing collective intelligence” with “getting better the more people use them.” They are not the same thing. The fact that my neighbor uses Google’s search engine, rather than Yahoo’s or Microsoft’s, does not increase the value of Google’s search engine to me, at least not in the way that my neighbor’s use of the telephone network or of Facebook would increase the value of those services to me. The network effect underpins and explains the value of the telephone network and Facebook; it does not underpin or explain the value of Google. (Indeed, if everyone other than myself stopped using Google’s search engine tomorrow, that would not decrease Google’s value to me as a user.)

Ah, I say to myself: Nick only sees first order network effects, what you might call endogamous networks, those that require the user to be part of the tribe. Thus, phone networks, and networks like Facebook. But the internet is an exogamous network; its benefits increase by the extent to which it reaches out to new groups, increases cross-breeding, and thus the total robustness and variety of the gene pool. This is why links matter, why web services matter, because they extend the reach of the network. Understanding the benefit of exogamous networks requires a more subtle calculus than Nick is applying. It’s not necessarily that you benefit directly from belonging, but the fact that you belong allows others to harvest the benefit of your participation.

Consider Google: The underlying network that Google is based on is one that they neither own nor control, the web itself. It has both endogamous end exogamous elements. No one controls it; its richness and diversity depends on that fact. And yet, there is a benefit to belonging. If there weren’t, sites would use their robots.txt file to tell Google and other search engines to stop spidering them.

Yes, you might say: but other search engines have access to that same network. And here, of course, is the first lesson: Google is better at spidering that network than their competitors. They thus benefit more powerfully from the network that we are all collectively building via our web publishing and cross-linking. Nick correctly points out that Google has built superior systems, and that these are the source of their competitive advantage. But that’s a diversion. Why did they build those superior systems? To harness the power that was hidden in the network more effectively than their competitors.

Google’s second network effect advantage is PageRank. As Robert Scoble so insightfully noted back in 2003, we contribute to Google with every link. Google realized that there was an additional layer of meaning hidden in the network. Far from being a contradiction to my network-effect hypothesis, as Nick claims, this is a validation of it. Advantage came to Google for seeing more deeply into the nature of the network, and building tools to harvest and apply data that was hidden in the network graph.

Google’s third (and most profitable) network effect insight was, of course, the ad auction. And once again, Nick misses the point. He says:

Now it’s true that, if you want to define market liquidity as a type of network effect, Google enjoys a strong network effect on the advertising side of its business (which is where it makes its money), but it would be a mistake to say that the advertising-side network effect has anything to do with Google’s dominance of the searches of web users.

It isn’t that the advertising-side network effect has anything to do with Google’s dominance of search, but rather, that Google’s dominance of search is central to the design of their ad auction. You see, while Yahoo! (nee Overture) sold keyword advertising to the highest bidder, Google realized that they could mine their users’ clickstream activity to predict which ads would be most likely to be clicked on, and by what ratio, and thus sell to the best combination of price and actual click through. Thus: higher revenue, more ability to invest in infrastructure, better results for advertisers and users, thus more users, thus better data, thus better results for both organic search and advertising (both of which do, in fact, matter to users, no matter what Nick thinks).

And of course, from there, you can also see other areas in which Google (and their competitors) are doing just this, from Google Docs and Spreadsheets (which exhibits the obvious kind of network effects that Nick is comfortable with), to mining clickstream data, to machine translation.

In short, Google is the ultimate network effects machine. “Harnessing collective intelligence” isn’t a different idea from network effects, as Nick argues. It is in fact the science of network effects – understanding and applying the implications of networks.

I want to emphasize one more point: the heart of my argument about Web 2.0 is that the network effects that matter today are network effects in data. My thought process (outlined in The Open Source Paradigm Shift and thenWhat is Web 2.0?, went something like this:

  1. The consequence of IBM’s design of a personal computer made out of commodity, off- the-shelf parts was to drive attractive margins out of hardware and into software, via Clayton Christensen’s “law of conservation of attractive profits.” Hardware became a low margin business; software became a very high margin business.

  2. Open source software and the standardized protocols of the Internet are doing the same thing to software. Margins will go down in software, but per the law of conservation of attractive profits, this means that they will go up somewhere else. Where?

  3. The next layer of attractive profits will accrue to companies that build data-backed applications in which the data gets better the more people use the system. This is what I’ve called Web 2.0.

It’s network effects (perhaps more simply described as virtuous circles) in data that ultimately matter, not network effects per se. Nick probably wouldn’t think of Nuance as a network-effects driven company, but it is, because their applications and services depend on data that gets better the more people use it (or have their data harvested in one way or another.) More speech in more circumstances and more domains makes Nuance better for the next user. No user thinks that Nuance is better because of them (and because many of Nuance’s products are standalone, this is in fact true.) Yet Nuance, like Google, has figured out how to harvest data contributed by millions to build a better product.

Nick also took exception to my characterization of Wikipedia as a network-effects driven success:

I would also take issue with O’Reilly’s suggestion that Wikipedia’s success derives mainly from the network effect; Wikipedia doesn’t become any more valuable to me if my neighbor starts using it. Wikipedia’s success is probably better explained in terms of scale and scope advantages, and perhaps even its nonprofit status, than in terms of the network effect.

How wrong can you be? If there weren’t a network effect driving Wikipedia, Knol and Citizendium would be succeeding. Wikipedia got there first, to be sure, but they also built an infrastructure and a workflow and a philosophy that recognized that the collective of all users was smarter than any expert, and that barriers to participation would slow down improvement in the data. There isn’t a Facebook-like benefit in “belonging” to Wikipedia, but the application understands something that its competitors don’t about harnessing the network and its users to improve its data.

In short, Facebook is the obvious network effect case study. But we learn more by studying what is not obvious: the way internet sites and companies have derived competitive advantage by leveraging different kinds of network effects, most particularly (but not exclusively) to improve the data on which their services are built.

I’m making no claim, as Nick seems to think, that there are no other levers of competitive advantage in the internet era. Nor am I claiming that every network-effect business will be more successful than those that are not, precisely because there are other levers of competitive advantage, but also because some markets are more monetizable than others. But I am claiming that there will be significant differences in profitability between companies that find a network-effect sweet spot in a lucrative market, and those who embrace the commodity end of the business. And sorry, Nick, but I consider the cloud infrastructure business to be the commodity end of the business. It will look like the web hosting business, say, with a bunch of large, capital intensive providers, and not like the hugely profitable company extracting monopoly rents that Hugh Macleod (whose post The Cloud’s Best-Kept Secret triggered my own) envisions. Monopoly rents, if they occur, will be at higher levels in the cloud stack.

tags: , , , , , , , ,
  • http://www.gandalf-lab.com/ Niraj J

    I think the point as I understand from nick’s blog is that network effect might not be the single strategic advantage to make money in cloud computing.

    Who says that x86 architecture will continue and that more hardware innovation can lead to a competitive advantage on the hardware side. who knows – products like Arista could be where the high margin business shifts. Or that AppEngine type properitery platforms with an Integrated Stack might be where margins are. Data could certainly be where the competitive advantage resides but dont expect all fortune 500 to allow companies like google to mine their data

  • http://www.gandalf-lab.com Niraj J

    on the x86 comment above:
    It is very likely that once the notion of computing in the cloud is established , the next wave of innovation could be in chipdesign. Once the commodity moves from computers to computing. bettering the computers could be competive advantage. Note that in the airline industry – GE with jet engines makes the most margins , then come the boings and the least margins are with the airline’s.

    You need to have a differnent appetitite and culture to be in the commodity business

  • http://twocroissants.wordpress.com Bertil Hatt

    I like your distinction between endogamous and exogamous effect, but — sorry to be the hair splitter here — the economic literature prefers to mention two-sided markets, economies of scale in matching and technological lock-in (exog.) that are distinct from both neighbourhood mimetism or club effect (endog.) The difference are crucial to decide whether the lock-in can allow an actor to sustain an abusive position, or what solution there is to a monopoly.

    More on my blog:

  • http://www.mymeemz.com Alex Tolley

    Niraj: The reason both GE and Boeing make profits is that they have almost no competitors due to the barriers to entry for these businesses. Conversely, airlines have many competitors and the barriers to entry are low for new airlines to enter. The main barrier to entry is gates at airport terminals, because airplanes are always available for lease.

    Telephone networks are the classic network business, and this paid of very well for AT&T until the rules were changed. Now the industry is trying to get those profits back by simply becoming a new monopoly – or rather a few local monopolies.

  • http://www.mymeemz.com Alex Tolley

    Tim: “I’m making no claim, as Nick seems to think, that there are no other levers of competitive advantage in the internet era. Nor am I claiming that every network-effect business will be more successful than those that are not, precisely because there are other levers of competitive advantage, but also because some markets are more monetizable than others. But I am claiming that there will be significant differences in profitability between companies that find a network-effect sweet spot in a lucrative market, and those who embrace the commodity end of the business.”

    But you do state:

    “The company that creates the right platform for network effects in data may well achieve the scale that Hugh Macleod envisioned.”

    Which implies that network effects will be stronger than other drivers, or that the other drivers will not operate effectively.

    I would hope that neither of these cases occurs, or that the regulatory framework will prevent that. (Maybe even open source in all its manifestations will help prevent concentration). It would be far better that we keep a competitive landscape so that traditional “lock in” monopolies that can extract super normal profits do not happen this time around. I also think that a more diverse ecosystem might be more robust from a single point of failure POV.

  • http://nessence.net Alex Leverington

    I agree with your analysis. Furthermore, the companies with effective margins will be capable of delivering at all levels of infrastructure, networking, and services and will have engineers, both social and technical, who understand how these systems interact, interface, and interfere with each other. Amazon, Google, and Apple are attacking all three. Atop all three layers are the actual value created for and consumed by us. The value is iPhone, Google search, and finding the book I want on Amazon and having it delivered the next day for $3.99. Whether these three companies have dominance over all three layers is not as relevant as their understanding and leverage of them. I would agree that “cloud computing” as an infrastructure, social graph, or service will not singularly yield high margins or present a monopoly. In fact, it’s quite the opposite. I feel with the globalization of economic systems and Internet’s lack of authority dictates a law such that any authority of ‘power’, with respect to power laws, will naturally be a dichotomy of being the most volatile and powerful entity of it’s domain. This doesn’t discount that fact that practical management and diversity are still imperative.

  • Anonymous
  • http://tim.oreilly.com Tim O'Reilly

    Mashable’s post about Azure and distributed computation provides some really important additional insights into layers of the cloud.

    It reminded me how, back in 2001, when I did my first “internet operating system” conference (long before we called it Web 2.0), I was trying to show how P2P, web services and distributed computation all together showed us something about the shape of the future, what the network could become.

    Since then, so much of the focus has been on the web end of the cloud. Mashable does a good job of reminding us that that’s not all there is.

  • Michael R. Bernstein

    Arguably, Amazon *has* discovered the right platform for network effects in data, simply by discounting intra-cloud bandwidth costs.

    If Company A and Company B want to create or run an application that operates on data from both companies, that application will be cheaper to run within a single cloud than between two clouds.

    The same argument applies to a company running all their own apps within a single cloud, and so on.

  • http://brand.co.in Norbert Mayer-Wittmann

    Hi Tim,

    I have great for you and your vision regarding collaborative development, and I also have great respect for Nick Carr’s insights on the life cycles of technological innovation towards commoditization — and I think both of you make valid arguments.

    In this case, however, regarding “network effects” I feel you are perhaps both wrong.

    My hunch (I really don’t know, because I have not been following every individual post about this) is that both of you (and indeed a large part of the so-called “IT sector”) have a very superficial notion of IT in general.

    Very few in the IT sector, for example would regard natural language as a “technology” (and in fact the *FUNDAMENTAL* technology upon which all IT products must be built).

    Cardiologists and accountants and auto mechanics and parents and children and students and stock brokers each speak different languages — and this is what matters. What is more: If I have a heart attack, I do not care what a car mechanic thinks I should do (nor any of the others).

    That is where Google’s one-size fits-all approach breaks down — at the intersection of what it means to be a human and what it means to be a machine (a piece of technology): Language.

    Noam Chomsky understands this — he was perhaps the “father” of this approach to information / communication / language. Other linguists, too have understood it, too — but Professor Chomsky was very focused on building a framework for understanding language as “human hardware”.

    In this vein, I strongly encourage you to review the following statement made by Jack Ma at the web 2.0 Summit almost 2 years ago:

    >> They came to me and they said: “Jack, can you help us to do something?” I said: “Well, why do you think you can be successful in China?” They said: “We have a brand!” I said: “Brand? No, you don’t have a brand in China — you have a brand in the States.”

    :) nmw

  • http://www.stickyness.de sascha

    “The last three years have fundamentally changed the way people understand their location and geography. Looking at interactive satellite imagery of our globe is now commonplace. The next three years will bring even more innovation, unleashing greater data and details allowing users to understand not only the greater planet around them but their own personal web of friends and locations that sit inside of it. Take the opportunity to explore these technologies, not only to learn more about planet Earth but also what your personal slice of it looks like – maybe soon in real-time.”


  • http://www.pjtrix.com/blawg/ PJ Cabrera

    Wikipedia doesn’t get any better the more people search it for content, but it certainly gets better the more people there are correcting content and adding new pages.

    Be careful walking among those trees, Nick, you might accidentally bump into a forest!