Aug 23

Tim O'Reilly

Tim O'Reilly

Search Engine Spam?

I just read Phil Ringnalda's comments claiming search engine spam by advertisers on O'Reilly sites. This was a bit of a shock to me. Since then, I've spent a bunch of time talking to people about Phil's complaints, looking into what we're doing and what I think we should do. It's clearly a complicated issue, and my opinion has changed a couple of times as I've gotten more information. Apologies for the length of this entry. I'm still in fact-finding mode, but wanted to share my process rather than waiting till I have a complete solution.

The facts as I have discovered them:

  • A number of O'Reilly Network sites, including www.oreillynet.com, xml.com, perl.com, and others (but not the oreilly.com corporate site) have been running a set of text ads for hotel search sites, in a block entitled "Travelling to a Tech Show?" in the lower left column of the nav bar. In addition, there is a small separate block just below that, labeled "site supported by..." or "Sponsored Links" containing links to one or two other sites, with ads such as Computer Community, or, less obviously targeted to the O'Reilly audience, Mortgage Refinancing or Health Insurance.

  • These appear to be legitimate ads, albeit not specifically targeted to the O'Reilly tech audience. There are no links to porn, drugs, gambling, or scams. In fact, as our ad manager noted, "some of those hotel prices look pretty good." The ads do in fact point to sites that provide the advertised service. (The one exception that I found in clicking through on the links was one to a site that was labeled Web Directory, and on first click appeared to be a directory, but on second click down into any category, simply contained ads for a book on search engine optimization. That one I'm clear about: it's a deceptive ad, and needs to come off the site right away. Another so-called Web Directory is indeed a directory, but the only content when you get to the bottom of each category is a set of Google Adsense advertisements for the category. (Question: if Google is opposed to this type of site, as many of those commenting on the issue claim, why is Google providing these ads?))

  • Many of the text ads on our sites are placed by a company called 3Genius, but many of them come from individual advertisers via our normal ad sales process. Our ad team apparently restricted the range of possible links to travel sites (which seemed plausibly relevant) and a couple of other areas, though affiliate sites such as Osdir.com, servlets.com, and linuxquestions.org, which are O'Reilly branded but which we do not own, have less restrictive policies, which is why you will see ads for cuban cigars or Jack Daniels on osdir.com.

  • Phil refers to the WordPress case discussed by Andy Baio back in March. WordPress was hiding non-visible links to advertiser sites on the WordPress site in order to drive up advertisers' PageRank without that being apparent to anyone. What we've been doing is different in the significant respect that the links we sell for advertisers are clearly visible on our site, with link keywords that match the content of the destination. There may be cases on other sites where hidden link farms are being used solely to game the search market, but on O'Reilly sites, these are all visible links -- just like any other paid text ads.

  • That being said, it's become clear to me on investigation that these folks are indeed paying us for our Google rank, and not just for clickthroughs. We just aren't targeted enough for their ads to be justified on a click-through basis. What's more, using Google's link: keyword to check for top links to these particular advertisers shows that the O'Reilly sites they advertise on are among their chief link sources. They aren't getting independent links from users. In short, these advertisers are using O'Reilly and other highly ranked sites who accept their advertising to improve their chances of being discovered via search engines, rather than in quest of direct click throughs (although those may also provide some value for their ad buy.)

  • Google has an authorized way for people to show up arbitrarily high on searches: i.e., to pay for relevant Adwords. However, nearly all of the terms used in these links are quite expensive. So advertising on a site with a high page rank instead of via Google Adwords is a way of arbitraging the relative cost of advertising on the two sites. However, it has a downside in terms of the search engine user experience. The ad shows up as a sponsored link on the originating site, but as a legitimate result in the search engine.
So there's the heart of the question: is it appropriate for a site to monetize its page rank as well as its page impressions?

It's pretty clear that the practice of "cloaking" -- that is, hiding links so that you're selling only the page rank -- is illegitimate. But what if someone pays you for a real ad, even if you know that they are paying you primarily because of your page rank rather than your targeted audience? As long as there's no deception as to the nature of the sponsored link, and a legitimate opportunity for click through, isn't this still an ad?

That leads to a whole nest of hard questions: Where are the boundaries between legitimate "search engine optimization" to help people find stuff that they will appreciate, and "search engine gaming", to inflate the rank of sites that are less useful? Whose responsibility is it to solve this problem? Should web sites turn away advertisers just because they are performing arbitrage on Google and other search engines? Or is it the search engine's responsibility to adjust their heuristics to counteract any attempts to game the system? Or both? Is it legitimate for a site to improve its own user experience by hosting small, well-paid and relatively inobtrusive text ads rather than the large banners and popups demanded by many advertisers if those ads lead to a worse user experience on search engines?

Long term, I'm pretty sure that supporting people who game search engines is not a good thing. The result will be that search engines are less able to reach their promise as an expression of the collective intelligence of the net. However, I'm not (yet) convinced that this is an open and shut case with regard to the ads that appear on our sites.

First off: consider the terms that are being bought: Rome Hotels, Phuket Hotels, Jack Daniels, Cuban Cigars. Not terribly relevant to programmers, but certainly not completely irrelevant. What's more, if you're searching for any of these things on Google or Yahoo!, you may not get the same results that you'd get if there were no advertisers trying to improve their search standings, but you will in fact still get meaningful links. (jackdaniels.com is still the top link for Jack Daniels, for instance.) It would also seem relatively easy for Google or Yahoo! to adjust their algorithms to demote sites that merely appear to be travel brokers instead of actual travel destinations if they think their user experience is being damaged.

Second, advertising in general is designed to get people to pay attention to things that they might not otherwise notice. Sometimes ads are effective, and sometimes they aren't. But we have to recognize that most forms of advertising, and not just this one, almost always detract from the user experience. But they are accepted by most people as a necessary evil because most of us recognize that developing content costs money, and we accept advertising in exchange for free content.

I do recognize that Google's preferred form of advertising -- context-relevant ads via Adwords -- is a real advance in making ads useful and targeted. However, at least so far, our experience has been that Adwords revenue will not even remotely make up for the other forms of advertising we carry on our sites. So our alternatives are to: a) convert the sites from advertising to subscription, b) continue to support them via advertising, or c) shut them down.

Simply put, we pay O'Reilly Network contributors for content, and we pay our staff to develop and maintain the sites. The money to pay those people comes from advertisers. Readers get the content for free, and advertisers pay for the chance to get those readers' attention. It's expensive to create a quality website with original technology content--many O'Reilly Network competitors have gone by the wayside in the past few years. I can assure you that we're not merely "a publishing empire trying to bring in a few more bucks," as one person commenting on Phil's blog claimed. Offering ad-supported content is not a hugely profitable business, and we're just as much "someone literally trying to pay a bill" as the small guys who Phil's commenter gave a free pass to on this issue.

In business and life, however, things are rarely simple, as Phil notes in his comments on "violent ambiguity." Net-net: I'm uncomfortable with these ads, and have tasked my team with coming up with an alternative as soon as possible. These ads are running under a long-term contract, and we'll think hard before renewing it. We'll also ask 3Genius to remove the links to the overtly deceptive ad that I discovered. However, if we were to shut off this type of advertising today, we'd also have to shutter many of the O'Reilly Network sites.

P.S. These ads have been running on O'Reilly Network sites for more than two years. They have not been recently added, as Phil claims. I don't know whether this is a good thing or a bad thing -- that we were among the first to be a target for search engine optimizers, not just recently joining the parade :-) I do know that now that it's become clear that this type of ad is a long term problem for the health of the net, we've got to find a way to wean ourselves from them.

tags:   | comments: 121   | Sphere It

Previous  |  Next

0 TrackBacks

TrackBack URL for this entry: http://blogs.oreilly.com/cgi-bin/mt/mt-t.cgi/4251

Comments: 121

  Matt Haughey [08.23.05 10:27 PM]

"These appear to be legitimate ads, albeit not specifically targeted to the O'Reilly tech audience."

Just to throw a wrinkle into an already complicated picture: these are ads aimed at search bots looking for relevant link text, so that the bots will follow the link and index the site. They are not designed for humans and not what I'd call a legitimate ad per se. That's the one-two punch of this kind of advertising -- they have figured out a way to get around Google's banning of linkfarm sites by spreading their advertising across a range of sites (achieving the same result as a linkfarm) but at the same time, they are loading up the bot with their descriptive keywords of choice.

In 1998, search engine gaming consisted of repeating a word 500 times in white-on-white text in the footer of a site, and putting 50 related words in meta tags.

In 2005, people pay highly pageranked sites to link to them using very specific keywords they would like associated with their site in searches.

The method is more subtle, but the goal is the same.

  Chris [08.23.05 11:05 PM]


Maybe you can add rel=nofollow to your links to prevent the flow of "reputation" (i.e. PageRank), and see if these advertisers hang around.

  Michael [08.23.05 11:37 PM]

It's good to hear that the intent on your side hasn't been malicious, though ignorance isn't usually accepted as an excuse.

Anyhow, I've written a post regarding some things I've been thinking about for a while regarding linking between sites.

  Andrew Wooster [08.23.05 11:47 PM]

Not to misrepresent these "legitimate ads", but I would think ruunning ads for Cuban cigars would be illegal in the United States, given that the product itself is illegal. It's not quite as bad as running ads hawking North Korean Taep'o-dongs, but, you know.

But then, I'm not a lawyer.

  Duncan Riley [08.23.05 11:50 PM]

I think you've been really hard done by over this whole thing, and before excluding these people, who as you note are doing nothing more than paying for advertising, consider that the people using this feature (and I'd note I'm not one of them) in the most are actually trying to do the right thing in paying for their ads which helps the rest of us out. If you start banning them you're only going to force more of them into comment spamming, spam blogs, and other "black-hat" things because they'll no longer have a legitimate way in which to advertise their sites because a few chardonnay socialists out there believe you shouldn't be advertising on your site and all content should be provided for free and without advertising. By all means look for alternatives but don't discount this advertising, for all we know these people are decent people trying to make a living just like you and me. Id also note to check out their link profiles in Google and notice that many of your competitors also have ads from the same people, as well as respected mainstream media sites such as the Washington Times (with one of the advertisers).

  Martey [08.24.05 12:23 AM]

In response to Duncan Riley's comment, I fail to see how a failure in implementing their advertising method (which I think is suspect to begin with) would "force" advertisers into spamming. Even if advertising like Adwords did not exist, there is no excuse for spamming. I believe this to be one of the essential rules of the Internet, and not one that just "chardonnay socialists" believe in. If the advertisers are really interested in promoting their website in a legitimate way, then Chris' suggestion of rel=nofollow should not stop them from continuing to advertise on the O'Reilly network of sites.

  Matt Cutts [08.24.05 12:28 AM]

Tim, if you'd like to talk about this, I'd be happy to give Google's perspective. I work at Google on preventing webspam, so I think it would be a good conversation if you have the time/cycles.


Matt Cutts

Google Software Engineer

  Robert Douglass [08.24.05 12:32 AM]

+1 for ref=nofollow. Ask your advertisers how much they're willing to pay for a link that has that in it.

On the other hand, I'd much rather have the O'Reilly Network sites available for free (since they rock) than to lose them or to have to pay a subscription. So +1 for keeping the links if no other viable business model can replace them.

  Dave [08.24.05 12:40 AM]

Well, advertisers buying ads on sites with high PageRank is just like advertisers buying commercial airtime on popular TV shows (or the Superbowl). I think it's a legitimate technique. Of course, you still have final say as to whether you want to run the ad.

  Enno [08.24.05 12:57 AM]

Love the guy who commented on cuban cigars being illegal in the US and that this meant it was OK to discriminate against the ad. Because as we all know, the entire internet community, both servers and surfers is in the US. Or at least the bit we care about right? ... Oh wait.

And for the record yes I know the issue is deeper, where are your servers, where are their servers, who are they selling to, to name just a few...

Now to wait for some new email from some US firm lamenting that Summer is almost over... (and I should re-subscribe to MAKE perhaps?) ;-)


  Brad Templeton [08.24.05 01:45 AM]

There are indeed more subtleties. For example, Tim, you and I have both been around a long time and have been selling sponsorships on web sites since well before Google existed. So on one hand it seems odd to suggest you should stop because Google has changed the motives of the sponsors.

On the other hand, you may want to limit how much you profit from that increased demand. rel=nofollow would certainly fix that, and of course most ad programs don't contribute to pagerank because they usually use a redirect-CGI to count the clickthroughs in any event. So you don't have to explicitly use rel=nofollow, just put in a click counter.

On the third hand, when the pagerank algorithm was developed, the idea behind it was to count a link as a vote for the page. You're just selling your vote.

There is an interesting heisenberg problem with the way search engines now treat links. The less you link out, the higher your own pagerank. I am sure that discourages sites from linking out that might have done so before. Me, I'm just lazy. :-)

  Jochen Topf [08.24.05 03:56 AM]

Its a long-standing journalistic tradition and standard to separate editorial content from advertising. In our times this should not only mean that a human can see the difference but that a computer can also see this unambiguously. For humans there is an "Advertising" announcement on top of every ad, translated for computers this would
suggest to not only use rel=nofollow but also tagging the content itself as an ad, maybe with or something like it.

  Peter Nixey [08.24.05 04:28 AM]


It strikes me that you may not have to worry about the moral side of things as the onus is going to be on Google to force your hand.

Look at things from Google's perspsective. They've got nothing to gain from your sponsors harvesting your page-rank. They do however have a lot to lose i.e. the quality of their search results for those harvest-terms.

So Google aren't happy because their valuable page rank is being resold around the web, what do they do? Answer: they limit the total sum of page rank that you are given - you're free to sell it if you like but in so doing, you reduce your own.

How would such a system work? Answer: they reduce your page rank when you link to sites that prove to be irrelevant.

You link to unrelated cheap-Phuket hotel sites and they reduce your page rank. Ah but this is unfair you cry - we need advertising, you can't penalise us for that!

True say Google which is why we're going to give you a "no-index" attribute for elements *within the page*. You put all of your sponsored advertising within a div that has a noindex=true attribute and we won't dilute your page rank.

You can continue to sell your page rank but it becomes a zero-sum game, every bit you sell becomes the property of the purchaser and is no longer yours to enjoy.

Its Google's job to fix the page-rank leak in their system and until they do, it would be actually be irresponsible for large reputable firms such as yourself not to bring it to their attention. All of us suffer from bad search results as a result of such a leak - the quicker they fix it, the better.

(Google, you heard it here first - don't even think of patenting ;)

  Phil Atio [08.24.05 04:28 AM]

Matt, if you feel inclined to "give Google's perspective," why not describe it in a comment here, rather than conveying it in a private conversation? I, for one, would be interested.

  Peter Nixey [08.24.05 04:38 AM]

Right, ok so clearly a load of other people have already made the point about rel=nofollow. Should have read more closely but the point remains that such a system can't hope to kick-off until Google starts doing some serious penalising for bad links.

  Hendry Jenson [08.24.05 04:47 AM]

I wonder about Google giving them the private advice. I don't think that's the norma practice. Instead they drop your page rank to nil and let you figure itout.

  Danno [08.24.05 04:48 AM]

What Pete Nixey said, you don't have to take the ads off the sites, but you should at least smack 'em one for being underhanded about the purpose of the ads.

My suggestion was going to involve IFrames and Robots.txt

  cmon [08.24.05 05:48 AM]

What's up Matt Cutts(Google Engineer) offering help to O'reilly. Hey Matt get a clue you probably have a lot of information that people would find useful. How about you get a weblog and let other folks in on the conversation.

  Rogers Cadenhead [08.24.05 05:59 AM]

Love the guy who commented on cuban cigars being illegal in the US and that this meant it was OK to discriminate against the ad. Because as we all know, the entire internet community, both servers and surfers is in the US. Or at least the bit we care about right?

O'Reilly is based in the U.S., so it has to follow U.S. law. I've had to reject several advertisers on my web sites for products that aren't legal to sell here, because the money you stand to make isn't worth the legal liability.

  Shelley [08.24.05 06:35 AM]

Matt, I also had these same type of links at my site, as mentioned in Phil's comments (though of course not Tim's post). We even brought up the issue of how all this works in Phil's comments.

So are you going to involve Phil in this conversation? A guy named Greg Yardley has also had some decent things to say about competition and links. You going to involve him, too? How about me?

Or is this going to be a backdoor conversation with the important O'Reilly folks and the important Google folks, and then you'll all come out and tell the rest of what we need to do, and what's right and wrong?

  Tom Radcliffe [08.24.05 06:40 AM]

All this suggests is that O'Reilly isn't charging enough for ads--the arbitrage aspect points clearly to the solution to the problem.

Sites with higher PageRank should be charging more for text ads. This will eventually deal with the problem, as there will be no relative benefit in engaging in this kind of behaviour. That won't eliminate the behaviour entirely, but it will create a (hopefully tolerable) equilibrium level.

Sometimes "the market will take care of it" really is a solution to a problem.

  Roger Benningfield [08.24.05 06:54 AM]

"...and we're just as much 'someone literally trying to pay a bill' as the small guys..."

Tim: So if you take those ads down tomorrow, your car will be repo'd next month? If that's the case, I sincerely apologize. I had no idea you were in such dire straits.

If it makes you feel any better, though, I was eventually persuaded to give you a pass as well. Whatever the intent of your advertising clients, *your* goal isn't to game Google, and you shouldn't be treated as if it is.

  Alex Thomson [08.24.05 07:13 AM]

Isn't this really about negative externalities?

If only O'Reilly readers suffered from diluted search results, the cost of the ads would be clear. However, these ads also affect the search results that non-O’Reilly readers receive. For them, these ads are a pure negative and O’Reilly Networks, not Google, is to blame.

I think that’s the problem. The people commenting here (me included) are probably getting a good deal. An outstanding online resource, some barely visible ads, and some search results that I know well enough to avoid. I think it’s a good compromise for me, but it’s also very selfish one. Get rid of them, but not because I want you to.

(And for those who think rel=nofollow will help? It won’t. If it’s not already in the advertising contract, it certainly will be soon.)

  Michael Moncur [08.24.05 07:31 AM]

I just wanted to mention that O'Reilly might want to consider the *cost* of this type of advertising--not in reputation, but in your own page rank.

Every bit of the PR that passes into the "cheap hotels" site does *not* pass into the other pages of xml.com. There are lots of minor pages in your sites that would show up as the top result in all sorts of "long tail" searches if their PR was higher, and you're giving that up in favor of ads. Also, as someone mentioned, Google might reduce the site's total PR or ability to pass it in order to deal with the spam.

It might be worthwhile, but I have a feeling the ad salespeople think they're selling "real estate" and don't realize they're selling off the chance for your own pages to get traffic. This is the reason I don't run this type of ads on my sites.

Incidentally, I just found out that my email to my editor at O'Reilly has been blocked by a spam filter, and the sysadmin told me unapologetically that "almost the entire IP space" of my host (the largest dedicated server provider in the US) has been blocked from reaching any O'Reilly address. This is the kind of hard-line approach to spam that I expect from O'Reilly, and while I may not agree with it, I respect it. I'm surprised to find their policy a bit softer as regards search spam.

  Bob Aman [08.24.05 07:35 AM]

Ok, guys. Be nice. Google people said something and now we're curious, but no need to chew them out. Not all conversations need to or should take place out in public. This is not a "backdoor conversation" in which Google and O'Reilly plot exclusive exciting sorts of things that you don't get to be a part of. I can think of a lot more rational and likely things that Matt would want to talk about with regards to this situation.

In the past, advertising has always been about getting as many eyeballs as possible (ads in newspapers, on tv, product placement, etc), in the hope that those eyeballs would translate into cash. Advertising hasn't exactly changed from that goal, but the methods certainly aren't exactly the same. Ultimately, I'm not certain the true problem is O'Reilly's. I have a feeling that in the end, the problem is Google's.

Most marketting people seem to see search engine results as advertising. The problem is that Google sees their sponsored links as the advertising, not the results themselves. Because the advertisers and Google have two completely different views of the search engine results, inevitably, there's going to be conflict. Google is necessarily concerned about the quality of the search results, because if the quality degrades, people switch to another search engine. Meanwhile, the advertisers are concerned, as always, about the eyeballs, and so they want to be as high up on the page as possible, because that's where the eyeballs are. Now, they could pay for sponsored links on Google's search results, but for the keywords they want, as others have pointed out, that can be pricey. Marketting's job is to get as many eyeballs as possible for the least amount of cash. And many (probably most) marketting departments aren't going to consider (or perhaps even care about) the possible repercussions for reputation that attempting to advertise within the search engine results themselves will have. So inevitably, we're going to have tension between Google and the advertisers (which is only complicated by the fact that it's the advertisers who pay for Google, not the end user). Google has to provide incentives for the advertisers to do the right thing. They've already provided incentive for middle men, the sites that host the advertising, to do so (Misbehave? 0 PageRank for you!). But they need to find a way to give advertisers an incentive to play nice. And it needs to be a positive incentive somehow, because advertisers won't give a crap about negative incentives.

Ultimately, O'Reilly is really an innocent middle-man (assuming they correct the problem somehow now that it's apparent). The fact that it took two years for the problem to become obvious (and even then, it was only obvious because a well-known blogger happened to notice it and become sufficiently irritated by it to write about it) should be proof enough that going after the middle man won't work in the long term.

One thing's for sure though: it's a non-trivial problem.

  Kerri [08.24.05 07:53 AM]

"Simply put, we pay O'Reilly Network contributors for content, and we pay our staff to develop and maintain the sites. The money to pay those people comes from advertisers. Readers get the content for free, and advertisers pay for the chance to get those readers' attention."

But it's not that simple, based on what you said. The advertisers do, in fact, pay to get those readers' attention, but they're also paying you for something that you point out is much more important to them -- giving them the tools to play Google's system to bump up their ranking. The fact that you've gone back and deemed some of the content unacceptable suggests that there is a line here -- it's OK to boost the ranking of some advertisers, and not of others.

If you know why the advertisers are advertising with you (to get your assistance in boosting their search-engine ranking), and you still accept their advertising, you are selling something much larger than your readers' attention. Making a conscious choice to allow this page-rank boosting for some and not others means that it is, indeed, a conscious choice to allow it for some. Is that 'right'? Who'm I to say?

It's sort of like being the guy who sells newspaper advertising in a well-respected publication. You can publish any ad you want in your paper. Based on your paper's circulation, you might have a captive audience that respects you. Some guy comes to your office and wants to buy an ad. He says, "I'm going to announce a rally on the lawn across from the Statehouse, and I'm going to say that it's a rally for child welfare -- making sure children have all they need for appropriate physical and educational development. But really, I'm just going to set up vendor booths selling products of interest to parents, and I want to advertise with you so that lots of people show up. And it's not completely dishonest -- some of these toys *are* educational!It's a great way to get parents to see what we sell."

Even if you know that the advertiser is using you to 'play' the system, whether the 'system' is parents or Google, the decision still rests completely with you. But you cannot deny that they are paying for more than your readers' attention. Advertisers are standing on your shoulders, and you're holding them up there. Of course, whether or not that's a good idea is your business decision to make.

  Brian Turner [08.24.05 08:19 AM]

The whole text link industry has been ballooning for some time - but my personal observation is that Google has been developing multiple methods for devaluing such text links anyway.

So as a short-term advertising method, it may bring in extra income - but you may find it becomes relatively harder to get advertisers over the long term.

You may want to look at highlighting the adverts in some way - sort of imitate the AdSense and Y! publisher format - and instead focus on delivering audience in terms of traffic figures.

I figure O'Reilly is going to be a very attractive place for many companies to advertise related-topic products and services, but of course there needs to be a balance of price to demand.

  Zwack [08.24.05 08:21 AM]

Re: The comments about Legal liability for the Cuban Cigar adverts.

If you read the Article you would see that the cuban cigar adverts are carried on "affiliated sites that are O'Reilly branded, but we do not own".

Surely this lets O'Reilly off the hook on this one as they don't own those sites and are probably not responsible for their contents.


  phil jones [08.24.05 08:59 AM]

+1 for the negative externality comment above.

Also, isn't accepting search engine optimization ads the same as accepting money for flyposting? Essentially, you're taking money for pasting adverts on someone else's (Google's) property. (That's why your "mere arbitrage" defence doesn't work.)

I'm surprised by the number of people who think this is Google's problem and responsibility to fix. It's not normally an argument in defence of flyposting that the problem belongs to the person who's fence gets flyposted.

  Danny Sullivan [08.24.05 09:05 AM]

Tim, if you want an easy solution, slap nofollow attributes on those links. Problem solved, as others have said. That ensure that for most of the major search engines, the links won't be giving credit to anyone.

You only recently got this option, by the way. It's still only months old.

As others have suggested, and as you suspect, some people may buy the links specifically because they want search ranking boost. So do nofollow, and they might end up going away. Then again, some of them may find that despite you being "untargeted," you still send enough traffic to make the links worthwhile. After all, despite the targeting Google AdSense does, it can still put untargeted ads on pages as well.

The bigger issue is whether you decide you should have the right to sell links that might help boost rankings. Yahoo has a directory. Being in that directory help Google and other search engines consider boosting a site. Yahoo charges $300 per year to be in that directory. If they can sell links, you might argue you have the right.

Similarly, I've written before about plenty of large web sites, newspapers and such, selling links just as you've done. They're making money off an economy that's developed. And as you said, you're wondering why you should somehow not be able to tap into that as well.

Ultimately, I suspect you'll go the nofollow route to avoid bad publicity. I also suspect you'll find some people still want to buy links despite this, since links, in the end, have long been sold simply because of the traffic they can bring to a site.

  Matt Cutts [08.24.05 09:31 AM]

As others have noted, if you're going to sell text links that pass reputation/PageRank, the way to do it is to add rel=nofollow to those links.

Tim points out that these these links have been sold for over two years. That's true. I've known about these O'Reilly links since at least 9/3/2003, and parts of perl.com, xml.com, etc. have not been trusted in terms of linkage for months and months. Remember that just because a site shows up for a "link:" command on Google does not mean that it passes PageRank, reputation, or anchortext.

Google's view on this is quite close to Phil Ringnalda's. Selling links muddies the quality of the web and makes it harder for many search engines (not just Google) to return relevant results. The rel=nofollow attribute is the correct answer: any site can sell links, but a search engine will be able to tell that the source site is not vouching for the destination page.

  Shelley [08.24.05 09:39 AM]

Folks who have brought up nofollow: can you not see how morally repugnant it is to take money from people to put links on your site, and then slap a nofollow on these?

Who are we gaming then? Is it because these SEOs are somehow the others, the outsiders and therefore fair game? While Google is part of us, and therefore should be protected, even from itself? Even from it's own algorithms that inspire gaming?

If it's 'illegitimate' to hide a SEO optimized and sponsored link, then why is it legitimate to then deny the company what they are paying for in good faith?

We have to accept responsibilty for our actions, and if one action is to accept sponsored links at our site, then we have to accept the bad PR and the rest that comes with this, but deliver on that from which we gain.

Some people say that this type of action hurts the web, but how is it any different when we link to a friend's site to help them out; or do not link to a site of interest, because we're pissed off at the person? All of these are actions to deliberately give a link or withhold it because of pagerank, and all are actions that webloggers do daily. We have already done massive damage to Google's algorithms, and we started with the very games we played, a few years back.

Anyone remember Google Bombing? How about the purposeful acts to bury the Christian Scientist's site?

Webloggers have been 'gaming' Google for years now, and finally Google came out with 'nofollow' as a way of getting us to police our own actions, because the company that can route around web linking of days gone by has never figured out how to route around us. And I'm not talking about Tim O'Reilly or me having sponsored links. I'm talking about webloggers.

So if you're all really concerned about 'gaming' Google, then you might want to consider getting rid of your blogrolls. Oh, and you should probably get rid of links in comments. And don't forget to link to other folks, regardless of pagerank. By the way, don't link Technorati tags anymore. And you need to stop with the Deli.cio.us sidebar links, too. And while we're at it...

Give me a week and I can list out a hundred different ways each and every one of us 'games' Google. Daily.

In the end, at least let us be honest with ourselves about what we do. If O'Reilly were to take links for pay and use nofollow, that to me would be the truly dispicable behavior. And you know something? For all that Tim O'Reilly and I don't agree, I bet this is one we would agree on, 100%.

  Jay Fienberg [08.24.05 09:40 AM]

I like and use Google, but:

I think we're in trouble when we start talking about Google like it's an inherent part of the public / free / open Internet.

O'Reilly needs to sell advertising, and advertisers are working with the Google-is-commericial ecosystem, not in an open-an-free-search-engine ecosystem.

rel=nofollow is mostly just part of the closed / commercial aspect of Google (and other commerical search companies) as well.

O'Reilly can choose to take a moral stance on how its advertisers' links appear in relation to how they might be affecting Google (because Google tries to do the right thing and produce something of public value). But, it seems to me that this is all native to the commerical web, i.e., commerce-driven rather than driven-by-openness.

  Ted Weinstein [08.24.05 09:48 AM]

Not to denigrate the thoughtful and fascinating conversation on this issue, but wouldn't having all ad clickthroughs go through a one line redirector script on your site solve the problem immediately?

  Ram [08.24.05 09:58 AM]

Just so everyone on page, Matt Cutts, Google Engineer do have a blog and it is kinda new, but, already has quite some stuff out there,


  Peter Nixey [08.24.05 10:05 AM]

I think that Jay has a good point. It is important that we don't treat Google as if it is the very definition of what is significant on the web.

Page rank is as close as we have to a meaninful measure of a page's significance but at the end of the day it's just an algorithmic approximation to the truth.

Shelley, there's nothing "morally repugnant" about putting nofollow on advertisers links. Selling links is just that, selling links and a market price will adjust to reflect whether or not those links endow page-rank. I do agree that it's reprehensible to misrepresent such a deal but there's no suggestion that that's what's currently happening.

Phil, why are you surprised that people think this is Google's problem? They design the page rank algorithm and the design of that algorithm is what's skewing advertising value.

Page rank is valuable because people trust it to give them quality search results. If it can be bought then those results are not quality and we have killed the goose.

While it can be bought there's nothing wrong with selling it but Google need to get their act together and fix it if they want to stay top-dog.

If Google becomes purchasable, it loses its functionality as a search engine and people go elsewhere (and the value of page rank becomes zero).

This is by definitition Google's problem but I'm confident that they can fix it.

  Shelley [08.24.05 10:12 AM]

Peter, I find 'nofollow' to be bad use of the web for all purposes.

Jay and others, you should check out Greg Yardley's writeup on the commercial aspects of Google and competition in a post titled I am not responsible for making Google Better. It's a good read. Oh, and a good use of a link.

  Dan Lyke [08.24.05 10:35 AM]

I just got back from a vacation that I planned and scheduled mostly through web research. One of the most difficult things about the research for that vacation was digging through all of the hotel and B&B consolidation sites and excluding them from my searches while trying to look for information.

In one case, I saw a price that looked reasonable, tried to book through one of those "services", it gave me an incomprehensible error message, but that led me to go deeper and actually find the hotel web site, which showed me that:

1. The price of the service was the price the hotel would have charged me anyways plus their service fee.

2. The hotel finder service gave me an incomprehensible error message rather than the perfectly reasonable one that the hotel site gave me.

Good ads provide a service to the reader. The only thing these companies do is make reservations harder to get, and make my hotel rooms more expensive.

I'll be adding 'rel="nofollow"' tags to all of the links to O'Reilly sites on my PageRank 6 weblog today. If you guys can come up with some advertisers who actually provide a service to me, and who add value to a transaction rather than simply leeching from it by dint of a superior ad budget, then maybe that'll change.

But the web is a commons, and we have to work together to keep it useful.

And, Shelley, as much as I respect your writing, if you're going to reduce the value of the web to me then I'm going to pull the value of my links to you.

  Peter Nixey [08.24.05 10:49 AM]


I read Greg's article and you're right it is an interesting article. He doesn't however deny that the problem exists, he simply states that it's not the problem of site owners.

Greg says that these companies are big mature, competing businesses and it's not our job to nurture them.

He's clearly right but at the same time, trading page rank is cutting off our nose to spite our face. Two years down the line, we've all made a quick buck from page rank and now we can't find anything on the search engines - zut.

This is the responsibility of the search
engines but there's nothing wrong with giving them a few tips along the way. I want search to become better not worse, I assume that's true for most others?

  Jim Provost [08.24.05 11:08 AM]

As far as I know, PageRank is really rather trivial as far as search engine results go.

But I could be completely wrong. Google will adjust, they have to.

  Shelley [08.24.05 11:23 AM]

Dan, then both O'Reilly and I will be sad to lose you as a reader. Because if you did read us and feel moved to respond to something we said, but either wouldn't link us or would annotate the links with nofollow, you are gaming Google. You are changing the value of the link based on something other than its relevancy. You are doing so for personal reasons, or even political reasons--but you are doing so.

  hfb [08.24.05 11:46 AM]

I've never understood the whole 'relevance' of the adwords stuff as these are the same advertisers who show tampon commercials in the 20 minutes or so of ads before a movie starts. Sure, a lot of people need tampons, but seeing a giant tampon while you're on a date or munching on popcorn probably isn't the most desireable context. The glow-in-the dark post-it notes with a Nokia ad pasted onto the back of the movie seats and the live actor toothpaste theatre in front of the screen pushed the boundaries of relevance, too. I don't think of personal hygiene when I go to the movies any more than I think of travel when I visit ORA web pages.

Everything could be thought of as relevant or made to be relevant in a marketing context I suspect.

  Dan Sanderson [08.24.05 12:45 PM]

What's wrong here is the gap between the endorsement of the advertiser by the content provider perceived by the reader, and that perceived by an influential relevance algorithm. An eyeball ad is obvious to the reader as an endorsement from the editor: It says the content provider is willing to associate itself with the advertiser by giving them prominent display space.

PageRank ads, on the other hand, are anonymous endorsement: the effects on the display space, and on the reader, are minimal, and the effect on Google's PageRank algorithm is not obviously affiliated with the editor providing the endorsement. In this arrangement, editors may be more willing to accept link placement from low quality advertisers, and that's definitely the case here.

If making the endorsements invisible to the reader is obviously wrong, then I don't see how tucking them away in the corner of a site below the fold is much better. If you've carried these sponsored links for two years, and only now are people noticing, then that's a problem. Only after two years of hosting PageRank ads is oreillynet.com being held accountable by its readers for these endorsements.

The "hard questions" about selling PageRank are irrelevant. These sponsored links could just as easily be made more attractive to the eye, more prominently displayed on the site, and be for a more reputable product, and still be intended, in part, to exploit PageRank. I don't think many of us would object to the ads under those circumstances. At that point, it's more obviously the search engine's responsibility to improve the quality of their results in the face of genuine endorsement of advertisers by highly ranked web sites.

  Neil Katin [08.24.05 01:05 PM]

Tim, I respect that you are grappling with a difficult issue, and your insight that you are selling both your page rank as well as the actual advertisement seems plausible.

It would probably be a difficult decision for you, but you could separate the two issues by using the rel=nofollow tag in your anchor tags to avoid lending your pagerank to your advertisers.

See preventing comment spam for more info.

  Ole Olson [08.24.05 01:15 PM]

Matt Cutt's wants you to use 'no follow' so that Google can provide better search results. He also has a vested interest in increasing Google's take on Adword sales and this is a nice customer self-service model for Google that doesn't force them to do anything.

O'Reilly should raise their ad prices. Some improvement in the O'Reilly ad sales equation (what is acceptable)is appropriate and part of the natural cycle. But until Google sees a revenue impact from the arbitrage opportunity nothing will happen.

  Chris Reuter [08.24.05 01:48 PM]

I think these sorts of ads are a bad thing for both O'Reilly Networks and for legitimate advertisers. Consider: if I happen to be looking for cheap hotel rates and I click on an ad, I expect to be taken to a legitimate travel website. If I find myself at a link farm instead, that reduces the chances that I'll click on an ad on that site next time, which is bad for all of the other advertisers.

These sort of pseudo-ads decrease the value of advertising on the website, which will lose you money in the long term.

On the other hand, if everyone knows that all of the ads are legitimate, it increases their credibility. That makes advertising there more valuable and you can charge more.

  Andrew Wooster [08.24.05 01:57 PM]

"""If you read the Article you would see that the cuban cigar adverts are carried on "affiliated sites that are O'Reilly branded, but we do not own".

Surely this lets O'Reilly off the hook on this one as they don't own those sites and are probably not responsible for their contents."""

I did read the article, and I'm pretty sure it would take a lawyer to determine if O'Reilly is legally liable here. Regardless, very few people get prosecuted for this, so criminal liability is besides the point. However, does O'Reilly want to continue to be associated with a business which is hawking illegal products by taking advantage of the O'Reilly name? Their answer to this, I think, will be much more interesting than whether or not they continue to associate with other value-subtracted advertisers.

  Matt Mickiewicz [08.24.05 02:10 PM]

So if I'm interperting Matt Cutt's comments here correctly, the advertisers who are paying for these links have been wasting their money for the past few months because O'Reilly.com's Websites have been degraded by Google so they do not pass on PageRank/reputation, etc.

This comment by Google, if read by your ad clients, should scare them off pretty quickly if indeed they are buying these text links to siphon off PageRank.

On an unrelated note, on our own Website for Web Developers, SitePoint, I made the concious decision to reject these type of ads several years ago, simply because they look unprofessional and degrade the quality of our web presence.

That's not saying the decision was easy. With 1.8 million Web Developers per month visiting our Site and (until the last update), a PageRank of 8, the decision has cost us tens of thousands of dollars. But ultimately, it was the right thing to do.

  rcjordan [08.24.05 02:27 PM]

tim, it's spam. (hi matt)

  Gordon Mohr [08.24.05 02:46 PM]

I'm surprised no one has mentioned that with Toolbar and other 'click trail' info that Google has, Google can vary the 'juice' that flows along any one outlink on a link-by-link basis, rather than just page- or site-bases.

Put a bunch of junk links in the margin of your site, and if none of your users follow them -- never mind any 'nofollow' link -- Google will know, and not count those links as strongly as other links on your page that do get organically followed.

That is, they don't just have the web link graph via crawling, they can also assign weights to all the edges of the graph based on demonstrated user behavior.

I don't know for sure that Google's doing this, but they have the necessary info, and they'd be silly not to exploit it.

(They could even weight the trails of those users that make them the most money -- people who disproportionately click on ads and actually patronize advertisers with real purchases -- moreso than the average browser, gently pushing all web traffic along the same paths that boost their bottom line...)

  laura [08.24.05 03:14 PM]

Q: "So there's the heart of the question: is it appropriate for a site to monetize its page rank as well as its page impressions?"
A: If you really think that the PageRank shown in your browser is tried and true. PageRank should be taken with a grain of salt. Pretty intersting that so much business is conducted around a toolbar icon that may or may not even really reflect Google's projected "importance" of a page.

But more importantly, legitimate links are legitimate links. Although it helps both the provider site, the advertiser's site, the provider site's users, and probably the search engines to link to and from pages with the same or similar TOPICS. A large part of SE algorithms seems to be determining the topic of a page by looking at the social networks they link to and from, and more specifically, the content of the pages in which they link to and from. Link to travel sites if you want to rank higher in the travel community, not if you want to rank higher for Perl scripts.

In the case of O'reilly sites, I would imagine they would be much better off screening the type of places they link to - legitimate, content-rich, on-topic websites & weed out those who are selling cuban cigars. Unless, of course, cuban cigars are phenominally popular among O'Reilly's readers...?

  jonblock [08.24.05 04:53 PM]

Perhaps I'm missing something, but it appears that Shelley is arguing that rel=nofollow would somehow undermine the advertiser's benefit by preventing any people from visiting the advertised page. I don't know how that tag works in detail, but I interpret it to mean that users can follow the link without a problem, while systems like Google's PageRank algorithm ignore the link altogether.

One important distinction comes from understanding what the advertiser is paying for and entitled to. If the advertiser is paying for impressions (as I always assumed to be the model), then there is no obligation to make that impression into a value-added link in the PageRank sense. If the PageRank benefit is built into the ad sale (honestly, with both sides of the transaction aware of the "game"), then the host would be obligated not to undermine the link with a rel=nofollow tag.

My take is that the secondary benefit to the advertisers (PageRank boosts) is *not* part of the host's intent with the ad space sale. And it certainly can't be argued that an ad link is intended to have the same weight as a bona-fide link-as-recommendation placed by an author within the body of a page. O'Reilly's site doesn't think the advertiser's page is worth linking to on its own merits. It merely presents a box for "your ad here," and that advertiser happens to fill the box at the moment. At least, that's how I as a user of Google's search engine want PageRank to interpret an ad link.

Having said all this, I must say that I don't think the rel=nofollow approach would always work. I don't play in this pool and don't know all the rules, but don't some ad systems, including AdSense ads, provide their own links within a larger block of content provided by something like JavaScript or a back-end web service connection? In other words, I imagine that O'Reilly and others might not be able to assign rel=nofollow to the advertisers' links even if they wanted to. Someone suggested marking a SPAN or DIV with a tag that means "this contains advertising". If the search engines agree, everything within such an element (including nested child links) may be ignored for PageRank purposes. That way, O'Reilly can sell ads to the highest bidders (subject to Cuban cigar issues) without the risk of undermining the PageRank system.

  Anna Haynes [08.24.05 08:37 PM]

> "Someone suggested marking a SPAN or DIV with a tag that means 'this contains advertising'"

jonblock (and 'someone'), that is an excellent idea.
(I was going to suggest a "rel=advertisement" attribute, but adding it to the SPAN or DIV would be easier for the host to enforce.)

What's fascinating to me, when these issues crop up, is discovering how different people are with respect to moral reasoning. Sometimes it's surprising.

  bowerbird [08.24.05 09:50 PM]

shelley said:

> Webloggers have been 'gaming' Google for years now

wow. a blogger came right out and said it. amazing...


  SamB [08.25.05 01:08 AM]

Honestly, I really don't understand why people are getting their knickers in a twist over this.
People pay O'Reilly for advertising - no-one complains. People pay O'Reilly for advertising with text links in order to help their search engine placement and suddenly the world is coming to an end.

Calling this sort of thing spam is stupid - spam clogs up email boxes raises the cost of Internet provision for all of us by necessitating more powerful email servers. Text links on websites do none of these things.

  Teddie [08.25.05 02:27 AM]

O'h the irony.

In a discussion about whether or not text links are spam, which includes Matt Cutts from Google; its ironic that there happens to be a Google Adwords advert on the right of the page, offering to sell text links to increase PageRank.

  Teddie [08.25.05 07:10 AM]

Darn it's gone now. But even funnier is that it now shows an advert for the Financial Times, the other major publisher who got in a text link wrangle a few months ago, and underneath that a Google Ad.

For more info see:

I don't know what you see in the US but from the UK these contextual shenanigans are hilarious. The main debate around the FT link was the fact it was hidden.

Are Google having fun and hinting at something?

  Tim O'Reilly [08.25.05 07:18 AM]

Teddie --

Interesting note (as are all these comments -- thanks everyone for all the great insight and help understanding this issue). Seems to me that if Google is so against this, they shouldn't be running ads for services that do this. But Google's big enough that one hand may not know what the other is doing, just as I discovered at O'Reilly, which is 1/200th of Google's size in terms of number of employees.

(Speaking of Google, Matt, would love to chat -- send me email on how to reach you.)

Unfortunately I'm travelling now, with limited connectivity, or I'd be responding to more of these great comments.

Meanwhile, one last thought on the ambiguity of this situation. As I've put pressure on our ad team to cut off this type of ad as soon as they can wean themselves from them, they've come back at me with this question: what about tech advertisers? There are apparently a number of tech advertisers who want these kinds of text links as well. Here the fog gets even greyer. Would love your collective thoughts on this issue.

  Graywolf [08.25.05 07:46 AM]

I think everyone should go out and put "no-follow" tags on all of the ads they sell. That will enable people who sell advertising without them to charge a much higher price. Supply and demand.

  Alexander S. [08.25.05 09:04 AM]

"Advertising in general is designed to get people to pay attention to things that they might not otherwise notice. Sometimes ads are effective, and sometimes they aren't."

Tim is overthinking this whole thing. If one replaces 'people' in the above quote with 'agents' where an agent may be an individual reading for edification, an employee doing research or a program scanning a page on behalf of a search engine, these ads are absolutely legitimate. They aren't aimed at the same target audience that Tim's sites are, but the advertiser is paying for exposure and that's what's being provided.

Now, if Tim hadn't entered into the business of selling advertising with mixed motives (he wants to make 'just enough' to support new content creation but doesn't want to degrade user experience) there wouldn't be any issue at all. Besides, arbitrage is a good thing. Its existence forces the big advertising networks to lower their prices and causes search engines to refine their algorithms.

  Ian McAnerin [08.25.05 09:26 AM]

I think there is some confusion between content links and advertisments. Slapping a nofollow on most content links (links placed in context of a post/article to give contextual, hypertexed information to the reader) undermines the web, IMO. The web works on links, not isolated nodes. An essential part of the commons are the links between related resources. That's what the web *is*.

But advertising is a different animal. The advertiser does not dictate the advertisment, the content provider dictates what they are willing to do in exchange for money.

Some ads could be completly relevant. Some publications have such stringent advertsing policies that ads on their site are often at least as relevant to their readers as the articles. Others sell out, or deliberately exert less control.

All a nofollow does is tell a search engine that you don't really vouch for a link. It's up to the engine to decide what to do with that information. It's entirely possible that a network that automatically nofollows everything would have it's links treated differently than one that nofollows only after careful thought.

If you are concerned about the commons (and I am) simply *think* about who you are linking to, and why. If you are providing contextual information that is helpful to your visitor, then link, and link properly (no redirects or nofollows). If it's there but you don't know why or can't vouch for it, then nofollow it.

I would personally nofollow comments in blogs automatically because I'm not the author and can't vouch for the quality of the links. Blog spam is bad, and one of the main reasons the nofollow tag exists. But I might make exceptions if I followed one and found it good. Then I can vouch for it, and then you can ride off my reputation (which is what link weight and PR is all about)

Re: "gaming" the engine. Choosing to not make a link clickable (or not linking at all) is no different than adding a nofollow, except a nofollow is more user friendly. The true "gaming" would be linking, or not linking, only for search engine considerations, rather than your audience - if you vouch for it as useful, link, if not, don't. Simple.

I would consider someone who "horded" PR by not linking out at all on their site to be gaming the engines just as much as someone who linked but nofollowed, or who linked without any editorial review.

The only qualification would be if you promised to link knowing part of the request was related to search engines and then "poisoned" it. That's just unethical.

Quick analogy: Let's say a friend asks you where to buy a new computer. Linking is like telling them that you prefer X, Y, or Z. Linking with a nofollow is like telling that "I haven't used them personally and don't know much about them, but I hear X, Y, and Z aren't bad"; and not linking at all is like saying "I don't recommend anyone" either because you really don't know, or because you want them to buy only from you and wish to limit their ability to evaluate their options.

Link to trustworthy sites, "pass on" unverified information with nofollow, and don't link to crap, and the web will probably be a better place for it.


  Daria Goetsch [08.25.05 11:00 AM]

Tech ads would certainly be on topic for O'Reilly Media. If you choose to sell ads you want those ads to be on topic for your website. The real question I would see being asked is do you want to sell text ads or graphic ads. The use of text keywords with a hyperlink in an ad are what people are looking for when buying an ad for PageRank purposes. Graphic ads are not going to have the same value to those looking to gain PageRank from ads. Researching more about link building and how it works might be helpful in making a decision of this kind.

  Dan [08.25.05 11:03 AM]

If your contract permits it, you could create a small script on your own site for all outgoing paid links (just 304 to the real url), and put that script in robots.txt. You can then put the real URL in a onMouseOver so that it shows in the bar correctly. If the idea of being a broker in the deal is counter to your goals, I believe you could also put the real url in a onClick="window.location..." to still permit a direct link without associating your pages with the ads and without totally breaking the experience for non-JS users.

Intermediate scripts are absolutely everywhere in the advertising business, so it's not unreasonable for you to implement them in your case, IMO. Generally as an ad skeptic (I personally would never start my travel search by clicking on some random ad on anyone's site), I would be more inclined to trust an advertising intermediate link if it was very short and the url had obvious and clear variables.

  Daria Goetsch [08.25.05 11:25 AM]

One more quick comment.

The quality of content on the O'Reilly Media website makes it an authority site. In link building terms this means other websites want to link to you for your content as a leader in the tech community. Keeping your ads within tech content would help maintain the quality of your authority site.

  3Genius - Arlen Ritchie [08.25.05 11:47 AM]

On behalf of 3Genius, I wanted to respond to some of the misinformation out there and offer our perspective in an effort to meaningfully contribute to this discussion.

First off, I dislike spam too. It clogs up my inbox and yes I’ve seen it mess with search results. Spam is bad for the net and sucks out precious resources from people and machines alike while adding no value whatsoever to our lives. As a netizen and in an effort to be a good-corporate citizen, I also feel that I have a corporate responsibility to protect and better the web. That’s one of the reasons I take the concerns raised on this thread very seriously.

For clarification: 3Genius does not operate any web directories and those sites mentioned by Tim are not owned or operated by 3Genius. 3Genius Travel is a bona-fide travel company that seeks to provide travel shoppers with the best technology tools, information, and access to inventory in order to match travelers with the right hotels – an otherwise difficult process in the offline world (and one that I believe we’ve made easier in the online world).

I think the logical place to start is to clarify the definition of spam. Some of the posts on this issue have suggested that any advertising is spam if it’s not 100% contextually relevant or if it seeks to raise its profile. However, if that definition were accurate, the vast majority of all advertising and promotion both online and offline would also need to be labeled spam as most advertisements are not related to their host content. For example, when was the last time a TV commercial, outdoor billboard, or radio spot you were exposed to had anything to do with the content it was interrupting? Probably nothing. Tampons, insurance, SUV’s, Coke, and fast food have nothing to do with “Everybody Loves Raymond", your drive home, or Howard Stern. They’re just advertising.

Ergo, I would suggest, and I’m sure I’m not alone, that to be spam, a website or ad must be something worse: deceptive, unwanted, valueless, or intrusive. In the email context, spam is when you open your inbox and you get repeated and unsolicited bulk offers for stuff you don’t want from companies you have no relationship with. In a search engine context, I think spam is a site that adds no value or claims to be something it’s not – fundamentally, an obstacle to finding the right information.

So if we agree that legitimate advertising need not be contextually relevant and that advertisers have a reasonable right to self-promotion, then what’s left to consider is whether or not the pages these links promote are deceptive or valueless.

Our sites offer something unique and add value to travelers looking for an alternative to the big guys. Not only have we collected tens of thousands of proprietary hotel reviews from actual hotels guests, we also maintain an independent consumer ratings system for hotels -- a valuable alternative to the standard and corporate-generated Diamond and Star ratings. And unlike other sites out there that let anyone post unqualified reviews, only actual hotel guests who have stayed at that hotel within the last few days can contribute to our guest ratings.

3Genius has also been a tech innovator in the online travel space. Ahead of all the big sites like Expedia, Travelocity and Orbitz, we were the first to meaningfully integrate Hotels and Google Maps (with real-time filtering to help searchers narrow their search). We were pushing the tech envelope with AJAX before it had a name. In fact, this very blog commended us for that AJAX integration only a month ago. And other industry sites thought it was newsworthy too.

Of course, like most people online these days, we do try to optimize our content for search engines and other directories that use spiders -- but this is never done deceptively. While spammers seek to get anybody to their site by any means necessary, we only seek to get legitimate hotel shoppers to our site. All of our sites are optimized specifically to the services that we offer and at no time have I felt that the appearance of one of our sites has degraded the quality of the search results in which they appear. A person searching for hotels in New York will, when they land on our page, likely find the information on hotels in New York for which they were looking (plus they’ll also find our unique content and search tools). And if they didn’t want a consolidator site but wanted to contact the hotel directly, calling Hotels-x's toll free number 1-800-359-5672 provides a natural-language IVR that’ll give them the hotel’s direct phone number free of charge. Moreover, we also let hotels post their official website URL on our site.

Those that advocate against all search engine optimization take an extreme position that is not only unrealistic but also not representative of Google’s own position. Google promotes proper search engine optimization and for good reason -- because it make search results better. Optimizers create more meaningful and accurate page titles, use descriptive headings, provide alternate pages for non-indexable content (e.g. Flash and/or dynamic content), and construct thoughtful and accurate link anchor text that actually makes the web more organized. Proper optimization makes the web easier to index, provides for a more relevant search experience, and ensures that people can actually find what they’re searching for.

3Genius has always had every intention of optimizing our sites within these boundaries – often what is called “whitehat SEO". Part of the confusion and different interpretations on what is and is not appropriate stems from the fact that Google’s published guidelines on this topic are ambiguous and unclear. Moreover, these guidelines are subjective and lack objective examples of what is and isn’t acceptable practices.

Matt -- if you have the time/cycles, I would welcome your feedback on anything we have done or are doing. If any of it is outside of Google’s reasonable guidelines, you have my guarantee that we will act to fix it immediately. You and anyone else that wants to contact me directly can do so at oreillyradar@3genius.com.

Our goal is to bring relevant travel content, unique ratings and advanced search interfaces to travelers’ attention. If we come up high in search engine result pages as a result of our marketing efforts, I truly believe that it’s to the searcher’s benefit because our content is both relevant and useful, and users would have otherwise not have had the chance to see it. I like to think that our new visual hotel search interface is pretty cool and unique but I welcome feedback. Overall, our strong conversion ratios confirm that we’re succeeding at satisfying our audience.

Like Tim said, we’ve been big supporters of O’Reilly for 2 years. Not only do we purchase O’Reilly publications like everyone else, we also sponsor the sites. While we’ve often considered that the links might not actually generate search engine benefit as Matt eluded to (notice that some of the sites don’t necessary do well in search engine results), we in fact do get click-thrus from O’Reilly visitors genuinely interested in hotels. Yes Phil, people who travel to tech shows do surf O’Reilly.

Tim – RE: “They aren't getting independent links from users." This is incorrect. There are a lot of sites within the community that voluntarily link to our sites. If O’Reilly links dominate the results when using Google’s “link:www.example.com", it’s a matter of how Google displays the results. Google has admitted it doesn’t show all the backlinks to a site


  pmac [08.25.05 12:07 PM]

nofollow is an attempt to try to stop comment spam not to tell Google what links are paid for or not. By putting a nofollow on those links Tim is gaming Google just as much as if he left them static imho.

When PR was king and an expensive paid link from a high PR site was enough to rocket a site to to the top of Google, the results were never better. If someone was going to pay big money for Google juice they were going to make damn sure they had a site that was relevant to the query or they were not going to make a return on their investment.

No such thing as search engine spam, just crappy algorithms.

  Michael Moncur [08.25.05 01:12 PM]

Arlen: We can argue all day about the definition of spam--which usually amounts to "I can't define it, but I know it when I see it"--but the big question is how Google defines spam.

Judging from Matt Cutts' comments above, indicating that the PR-passing abilities of some of the O'Reilly sites have already been restricted, it's pretty clear that they define it as spam, and with Google's position in the market, it's probably bad business to do something that causes Google to penalize your site.

  graywolf [08.25.05 02:24 PM]

Google is a publicly traded company who's goal is to make a profit for it's share holders. They do this by selling advertising space.

They are not a regulatory body, branch of the government, law enforcement, or other institution whose goal is to serve the public's intrests.

Just a brief reminder for those of you who forgot.

  Gordie [08.25.05 02:48 PM]

I think Google should start being more specific as to what's allowed / not allowed. Why is Matt saying this for the first time in some random thread? If they don't like PR-passing or other SEO tactics, they should say so publicly. Once they've made a thumbs-up / thumbs-down decision, they should let the public know.

I've read the materials on their site and those materials are no help whatsoever to webmasters. Why be vague and leave people guessing? That only increases their oversight work for people who would otherwise comply, IMHO. They should set out the "law" in a policy doc for everyone to read.

  jeff hudson [08.25.05 03:25 PM]

O'Rielly is doing nothing but selling advertising space. If it isn't targeted or relevant, that's the advertisers problem. Yes, it would be nice for O'Rielly to provide their users with the most relevant ads possible, but it's clearly their decision, and it's a business decision, not a democratic one. He shouldn't have to alienate his advertisers just because he's passing along PR. If anything, he should take advantage of his PR and charge higher prices if he can get it. Have you people ever heard of a free market? You're in it now.

  aaron wall [08.25.05 04:54 PM]

The people who say "raise the price to weed out the off topic ads" fail to realize that high prices (above stuff that shows a solid ROI) are probably what weeded out the on topic advertisers in favor of the Cuban cigars.

When you sell expensive stuff links from an off topic authoritative site like XML.com can pack a punch in the search results, at least until the search engines discount those links. After you get used to the high prices some of the off topic sites are willing to pay for ad space it gets kinda hard to want to lower the price to try to make it look appealing to on topic sites. And after off topic ads are in place you may have to make the ad prices exceptionally lower than their value to get the right on topic sites to want to buy because the off topic ads lower the perception of value in many minds.

As far as that one web directory being deceptive, most directories start out fairly empty. Good ones eventually add a bunch of links to their database to ensure it is useful to directory users and search engines alike.

The fact that they have few listings would likely indicate that they are new, or being somewhat lazy, or wanted to get paid for every listing, or do not understand the directory business model.

I don't think having footer links is any more deceptive than selling Cuban cigar links.

BTW that looks afs though that person running the directory signed up or an affiliate account on my site & liked my ebook.

Personally tracking all of my affiliate sales myself I can assure you that your ad costed more than they are making in affiliate commissions from that link pointing at my site.

  aaron wall [08.25.05 04:57 PM]

>BTW that looks afs though that person running the directory signed up or an affiliate account on my site & liked my ebook.

I must have dragged & dropped the f there while re reading / editing my post. arg. It should read:

BTW that looks as though that person running the directory signed up for an affiliate account on my site & liked my ebook.

  Dan Sanderson [08.25.05 06:20 PM]

Arlen: On the definition of "spam", I don't think you're doing this on purpose, but you're confusing the issue by conflating definitions. E-mail spam is unsolicited advertisement sent to an e-mail address (and may also include scam e-mail and phishing, depending on who you're talking to). That has nothing to do with this conversation.

"Blog comment spam", "wiki spam", and the like are attempts to gain PageRank by exploiting web mechanisms that automatically post third-party content without review or endorsement by the owner of the site. Almost always, such attempts involve posting material irrelevant and intrusive to the site hosting the content. We're not talking about this either, though this is a related category.

"Search engine spam" is any attempt to affect the results of a search engine using any means other than putting valuable and term-relevant content on the web. These sponsored links are included in this category because there is nothing about the fact that the link is on the page that implies the linked site would be considered more relevant by people searching for those terms.

To Shelley and others discussing the use of "nofollow", a clarification: I believe nofollow was mentioned in the context of this conversation as a mind experiment. It is a hypothetical test of the intent of the sponsored links: If the sponsor would not allow their links to be tagged with nofollow, then it is clear that it is PageRank they are after, and the link is in the "search engine spam" category.

Assuming that search engine spam is actually effective at manipulating search results-- and we assume it is because someone seems willing to pay for placement-- then regardless of the sponsor or the terms used in the link, all sponsored links reduce the quality of the search data we all use to navigate the web. This would include tech-related sponsored links on a tech web site.

One more thing to consider: Say a site called SEOMaster.com wanted to run a text ad that read: "SEOMaster.com tells you everything you need to know about search engine optimization." If "search engine optimization" were a link to www.seomaster.com, would that be search engine spam? What if only "SEOMaster.com" was the link? It seems the former case is far more offensive than the latter, yet both bestow some PageRank and affect some search results.

Fanning the flames here are other tactics these particular sponsors in question are using to influence search engine results. There is nothing about the fact that someone owns a domain name that contains the words "rome" and "hotels" that implies the site would be more relevant to someone searching for "rome hotels" than other sites. A domain name like "rome-hotels-x.com" hints that the domain was chosen solely to influence PageRank. We might further assume that the site is not actually providing a valuable service, but is just trying to scam off PageRank, such as for ad impressions, or even to attract traffic for more nefarious purposes, and is therefore especially undeserving of inflated PageRank.

Whether it is or not doesn't matter, we've seen enough misrepresentation on the web by now that any attempt to manipulate search engines is in very bad taste. oreillynet.com's participation in the affair is similarly offensive.

  Arthur Bebak [08.25.05 08:14 PM]

As another old timer on the Net (Usenet days - Brad may remember :) my advice to Tim is to go to paid subscriptions. Netsurfer Digest did this years ago (the oldest web zine on the web, BTW - the kids call'em blogs now LOL!) and we've never regretted it. It's amazingly liberating not having to pander to irrelevant advertising or spend the money/effort chasing ad sales. And if you want to keep ads you can keep them relevant and your readers will appreciate you for it.

O'Reilly has first rate content and would have no trouble signing up tons of subscribers - say one $30/year sub for all your sites. It would give you a large revenue boost too, far larger then any ad revenue you loose. That's offset a bit by having to keep ahead of subscriber churn (through ads ironically), but not much, and it's a well understood business problem. Your Safari Bookshelf numbers should give you a small indication of what kind of subscriber numbers you could expect (I say small because Safari is relatively expensive).

Granted, this does fly in the face of a desire to give free and open content to the community. Philosophy collides with business here, but nobody wins if O'Reilly drastically downsizes their excellent content or is overrun with spammy ads which turn off readers.

But let's put all this in perspective. Some fraction of the community will be offended no matter what you do. Right now there may be some number of ads which game page rank on O'Reilly sites, but come on, it's not that big a deal. The content itself and the very reasonable ad/content ratio far outweigh any minor annoyance. The Google side of the issue, that is, the pollution of search results by commercial spam, is far more important. I must confess that I'm not sure that technological solutions will win that particular arms race.

  3Genius - Arlen Ritchie [08.25.05 08:35 PM]

Dan: I understand your points and acknowledge that some webmasters do unfortunately focus solely on garnering PageRank while failing to pay any attention to their content. However, your assumptions that Rome-Hotels-X.com is one of these sites and "is not actually providing a valuable service, but is just trying to scam off PageRank" are not only incorrect, they're impossible. If you look">look">look">http://rome-hotels-x.com/robots.txt">look closer, you'll notice that this site uses the Robots.txt">Robots.txt">Robots.txt">http://www.robotstxt.org/wc/exclusion.html">Robots.txt Exclusion Protocol and is therefore not able to be indexed by Google, nor able to accumulate Google PageRank. The exclusion parameters employed are not new and were used precisely to prevent anyone from harboring misguided assumptions about our intentions.

  Michael Moncur [08.25.05 08:41 PM]

Wow, Netsurfer Digest brings back fond memories. On e of my site's big breaks was a link in NSD back in 1995...

I for one would gladly pay for O'Reilly's content. It works for Safari, why not XML.COM?

  King Bastard [08.25.05 09:37 PM]

They've gotta make money somehow, right?

If you don't like it, don't visit the site. Or volunteer to pay for a subscription.

  Sara [08.25.05 11:25 PM]

This situation is all blown up. So people want to advertise on O'rielly sites. Is advertising such a bad thing? As a reader the web directories are on-topic but the hotels and cigars are not so much.

My husband and I found out about the Site Shift directory from this site. It is not a bad directory and has plenty of sites listed in the categories I saw. Without the advertising this site paid for we probably would of not found out about that site.


  Marcela [08.25.05 11:33 PM]

Frankly, how is this different from 50% of the web using Adsense? It's just a different form of advertising.

Does the fact that it's relevant means it better? Why would that be the case? What's so different from banner advertising - an accepted method of non-relevant advertising? Yahoo still uses this in their email. I understand banner advertising isn't prevalent now....but not because it was evil or wrong, but because it's not as effective.

So why is adsense okay but everything else is evil?

Everybody wants to make a buck...this is a capitalist society. Why fault people for trying to do what Google, Yahoo, everybody else is working very hard to do - monetize?

The web is an open medium and part of a capitalist society - why is something wrong when nowhere is it mentioned in Google's SERP's that it's against their TOS to engage in this method of advertising?

I just keep coming back to adsense..why is that form of advertising okay and not frowned upon?

  aaron wall [08.26.05 12:41 AM]

As a suggestion to the subscription model idea, I like what Danny Sullivan at SEW does. Most content is available free (in multiple formats - having forums, podcasts, articles, & a blog), but occassionally he offers subscription exclusive or longer or more in depth type articles to those who subscribe to SEW.

Most peole are not going to want to read a 15 to 30 page article about a single topic, but those who are really into a topic would gladly pay to have access to that extended coverage.

I would not suggest making all content subscription only, or someone else will work their way into your current market position.

John Battelle had a good piece on the pull to point concept (on why going completely subscription only and locking out all other access is bad bad bad)

If the cost of content creation is getting too steep there has to be some way to leverage the open source idea toward content creation as well.

There are many writers who would love the chance to write an article for O'Reilly. You could even make portions of the ideas / articles peer reviewed in some ways? Something like Slashdot but more in depth and cooler?

Heck I get a kick out of the fact that you mentioned one of my affiliates on your site. :)

Another option might be making your extended bits and article stubs that did not make your sites become bonus premium subscriber only content. There are so many things you could do with your killer strong brand...

  Crew [08.26.05 12:44 AM]

From Arlen/3Genius:

"that to be spam, a website or ad must be something worse: deceptive, unwanted, valueless, or intrusive. "

If I search for "Rome hotels" and your site comes up number 1 mostly because of these paid-for links and I visit your site, then that what you've done *is* unwanted and deceptive to me...
because I search Google under the assumption that their results are not paid-for.

  Frank D. [08.26.05 01:18 AM]

I guess I found more spammers http://www.csmonitor.com/

Look at the left hand side of this site.

  Frank D. [08.26.05 03:24 AM]

Tim, Your an idiot! Why do you think it's OK to link to health insurance sites and not to a web directory? I bet most of your users already have health insurance anyway. How many of them do you think really click on health insurance as oppossed to a web directory.

BTW, you have a lot more spammy links here:


Was there a special on spammy links on that site?

  Eric Giguere [08.26.05 06:47 AM]

Going paid-subscription-only is a bad idea if you want to reach a broad audience, and I think it only works if you're the exclusive source of some unique/high-value content. While O'Reilly is a source, it's not the exclusive source and other sites would just step in to fill in the gap with free content. Aaron's comment about premium customers getting extra stuff would be the best route.

When I'm searching for something with Google, for example, I'll occasionally come across a link to a subscriber-only site. Most of those are free, but they still require registration. But do I register/subscribe? No, I just hit the back button and move to the next item in the search results list. "Nothing to see here, folks, move on."

Yes, this points to the need for a "universal" subscription or some other form of micropayment system (which is essentially what a subscription is, an aggregated micropayment system) but as I argue in a comment to Tim's next blog entry, I think the "subscription-only" Web will have its own problems in the future anyhow.

  rogerd [08.26.05 06:56 AM]

When a publisher who provides free and useful content to visitors needs to screen advertisers against a hypothetical "Will Google think this ad is off-topic?" filter, we've yielded a bit too much power to the search engines. I've spent time in print publishing (where PageRank means a back cover is better than an inside page), and have found that a popular publication attracts lots of advertising that appears to be "off topic". Most publishers need these ads to stay in business, and will have their sales staff pursue any niche that seems to have the inclination to advertise and the budget to do it.

If Google is concerned that their algorithm can be gamed by hotel ads on a tech site, then perhaps the algorithm needs to be improved.

  Darren McLaughlin [08.26.05 08:25 AM]

These link should be no-follwed. As stated by others, the links are off-topic and don't help the general internet or the particular readers of your network of websites.

  dean [08.26.05 08:27 AM]

Why is Google indexing ads in the first place? Targetted or not, ads are not web site content, so they should not be indexed by the search engines.

As for web site owners, they are entitled to sell their ads to whomever they want. It's their business.

  Eric Giguere [08.26.05 08:37 AM]

Google can't reliably or easily determine that a link is an ad, especially since the definition of what constitutes an "ad" isn't fixed in stone. And some people find the ads relevant, to boot. Remember the news report a while back that interviewed someone who actually read and responded to unsolicited emails? What's spam for you may be paté for someone else...

  Patrick [08.26.05 10:11 AM]

Mr. O'Reilly - from what I have seen, you are doing nothing wrong.

When people sell a text link advertisement that is virtually hidden, that is questionable. If they sell a link that is hidden (as in the same color as the background) - then it's more than questionable, it's just wrong. You are doing neither of these things. When people sell the type of text link advertising that you are selling, there is nothing wrong with that. There are a few basic, common sense rules that must be applied. For instance, you don't want to sell text link ads that are misleading, such as selling a text link advertisement for "Sports" to nosportshere.com. You don't want to sell text link advertisements to a site that misleads people once they are at that site, either. You want to sell text link advertisements to a site that offers something, even if it is neither unique nor interesting to you. Even affiliate sites offer something.

But, text link advertising is an understated method of advertising that fits in with any design and does not annoy users. I don't serve popups, I don't serve interstitials, I don't serve invues, I don't serve popunders, I disable any noisy ads my networks serve, I disable any ads that I find inappropriate for my userbase, I do not load pages with ads. Without text link advertisements, my situation would be a lot different. Text link advertising has supported good sites and helped keep them online.

  PhilC [08.26.05 11:45 AM]

Matt Cutts wrote, "Selling links muddies the quality of the web ..."

Sorry, Matt, but that's nonsense. Selling ad links was all over the Web before Google came along and decided to base their rankings on links.

Matt Cutts continued, "... and makes it harder for many search engines (not just Google) to return relevant results."

That's true, but it's not the fault of selling or buying advertising links - it's the fault of those engines that base their rankings on links. And it is not for web people to fix the problem that the engines have - it's for the engines to sort their own problems out.

Let's be clear about it. There was no such problem before Google came along and based their rankings on links. Not only that, but they taught the other big engines how to do it. Any problem that they now have with links that are not genuine votes is of their own making - Google created it. So please don't try telling website owners how to link from their sites, Matt. That's not Google's concern. Google's concern is only how Google deals with links that it doesn't want to count. It's an internal problem, and nothing to do with other websites.

I doubt that many people would mind if Google algorithmically devalues non-voting links, even if it means devaluing some innocent ones. That's where you need to look for the answers to your problems - not at website owners.

The very idea that selling links "muddies the Web" is too arrogant for words. It "muddies" Google and other links-based engines, but it does nothing detrimental to the Web. When the search engines become the Web, instead of merely the signposts, then you can say that selling links muddies the Web.

  GoogleMuddiesTheWeb [08.26.05 02:00 PM]

The only thing that "muddies the web" is Google allowing every webmaster on the face earth continue to create made for Adsense sites.

  John S. [08.26.05 02:49 PM]

O'Reilly is loosing more creditability letting Google tell them what they can and cannot do.

Here is the secret conversation Tim is having with Matt C.

"We will do whatever you ask." say Tim

"Of course you will we are Google." Matt C. laughs with evil voice.

"We will link only to http://www.google.com" says Tim

"That is the only approved place you may link to" Matt explains.

Matt C. goes on "We make all the rules and they change from day to day. You better use an RSS feed to get the latest rules. Remember we are Google and you have to do what we say."

"Thanks Matt your my friend." Tim comments

"BTW Tim we make exceptions for big sites like yours so when you screw up my just give you a slap on the wrist. If you were a smaller site then we would of banned you with no questions asks." Matt C. explains with an evil laugh

Google has gone too far this time. Keep it up Google and the market share will keep slipping to Yahoo.

  RealityCheque [08.26.05 04:00 PM]

like a landslide.

Everyone might as well reset there homepages back to Yahoo! now.

  John S. [08.26.05 04:15 PM]

"like a landslide.

Everyone might as well reset there homepages back to Yahoo! now."

What a great idea. Let's all start a boycott of Google. Why should they tell people who they can and cannot link to.

I just change my homepage to Yahoo. I also cannot wait for the YPN. Google is bitting the hand that feeds it.

  lilia [08.26.05 06:03 PM]

To me it seems silly to expect Webmasters to act against their economic best interest in order to ensure that Google's search engine results stay accurate (translation: to ensure that Google's market cap continues to grow.)

A capitalist society simply won't work that way.

Especially when the behavior that Google is trying to discourage can be as benign as an inobtrusive, on topic sponsorship pointing to a quality site.

As others have noted, text link sponsorships predate Google, and are welcomed by users (unlike flashing banners, pop-ups, etc.)

I don't at all blame Google for trying to preserve that market cap and profit margin by trying to discourage Webmasters from behaviors that could jeapordize their market leadership position. This too is capitalism at work.

I just don't expect these tactics to work.

I think what we're really seeing here is an opportunity for a search engine that is able to align their ranking algorithms with the interests of the user bases of the sites that they index.

For this search engine, sites that provide a superior user experience will rank well, and sites that don't, won't.

Any search engine that links their future success to the willingness of webmasters to choose the interests of a mega, for-profit company over the interests of their user base would seem to be in a precarious position.

  Search Engines Web [08.26.05 06:06 PM]

There should have been more respect shown to those Advertisers - than to PUBLICALLY - express doubt about their intensions :-(

They are doing US a favor - by keeping these valuable resources: UPDATED, HIGH QUALITY & FREE - FREE - FREE.

Their motives are none of anyone's business - no one is being forced to click on any of those links in the OBVIOUS gray boxes.

The fact that they are being HELPED in their Google Rankings - may be an important business decision in deciding where to spend their valuable cash.

Anyone would want to stretch their dollars by using thse resources that provide the most "BANG FOR THE BUCK"!

It is much more preferable to have those Ads, than to have any loss in quality (no matter how small) in ANY of the O'Reilly sites, due to financial contraints!!

Googles' new patent - (contributed in part by Matt Cutts) - will show preferences to (Link) Theme Harmony

Information retrieval based on historical data

  John S. [08.26.05 06:56 PM]

"There should have been more respect shown to those Advertisers - than to PUBLICALLY - express doubt about their intensions :-("

Great point! Is this how O'Reilly treats it's business partners. The funny thing is they are just shifting the blame off of them.

Tim- own up and stop treating your business partners like crap. It's your company take responsility and treat people/companies with respect.

  Brad [08.28.05 04:32 AM]

Those advertisers were paying Mr. O'Reilly in good faith. I assume the checks didn't bounce. They were his customers and he just publically fed them to the dogs. I'm sure there would have been a nice quiet professional way to sever a business relationship.

As for Google. Off topic links are no crime nor should they be made into one. Look at television or your newspaper ads do they all match the topic of the page they appear on? Google's on topic nonsense flies in the face of several hundred years of real world advertising experience. Now excuse me while I go tear down that barn with the Mail Pouch ad painted on the side - it's off topic.

  Ed Johnston [08.28.05 07:48 PM]

In his comment above, Matt Cutts seems to be hinting that Google is not taking the controversial O'Reilly links at face value, even without nofollow.

Matt Cutts wrote:

"I've known about these O'Reilly links since at least 9/3/2003, and parts of perl.com, xml.com, etc. have not been trusted in terms of linkage for months and months. Remember that just because a site shows up for a "link:" command on Google does not mean that it passes PageRank, reputation, or anchortext."

  Stephan Spencer [08.29.05 04:13 AM]

That's more than just a hint from Matt Cutts. He's quite clearly admitting that they decreased the voting power of O’Reilly sites like perl.com and xml.com and downgraded the reputation value of some of their outbound links. And if you don’t want your site to suffer the same fate, you’d better tag your link ads with rel=nofollow so they don’t gain any PageRank. How do you like them eggs!

To me, that doesn’t seem quite fair to website owners. They work hard to build a content-rich destination site with good PageRank score. Google is diminishing their earning ability by insisting they cut off the flow of PageRank with a nofollow, thus decreasing the value of the link ads to the advertiser and ultimately the revenue likely to realized from that advertiser. Granted, you don’t buy links merely for PageRank, but of course it figures into the equation.

The problem lies in which link ads to vouch for. If I were the advertising manager for DailyItem.com, I certainly would not vouch for the advertiser of “Discount Vacations”, as the link points to a “doorway page” operated by Orbitz that links to a whole pile of other doorway pages (tsk tsk! Google warns against using doorway pages); on the other hand, I would vouch for the “Dancewear” advertiser, since that’s the company’s name and the link points to the home page of their ecommerce site.

Google should give the website owner the option of vouching for some of their advertisers without demoting their site. A black-or-white approach just isn’t practical here.

  Brian Davison [08.29.05 06:45 AM]

For many years now the web has been too large for search engines to ingore the information present in hypertext links. However, the mixture of links that are created has changed over the eight years or so since PageRank was formulated. Most links are no longer created by hobbyists; most links today are created by someone with an agenda (usually commercial), and so extracting (unbiased) information from them is now much more difficult.

Every decent search engine today has a mechanism to determine when to discount (or trust) the presence of a link; Matt's comments just give us a hint at what Google might be doing about it. Other engines may be taking a different approach. As the web continues to change, more research is needed to find better ranking algorithms.

While I believe Tim has a moral decision to make, in the short term it will not change how search engines value links, nor the buying, selling, and trading of links on millions of other sites. In the longer term, however, discussions like this are valuable and my hope is that they may lead to more transparent linking (where the intent is obvious).

  Brian Turner [08.29.05 01:47 PM]

"Another so-called Web Directory is indeed a directory, but the only content when you get to the bottom of each category is a set of Google Adsense advertisements for the category."

When I look there, I see listings under the AdSense.

  Dan Sanderson [08.29.05 03:50 PM]

This thread is just about over, but I'm intrigued by a direction this conversation has taken.

So far, nobody (except perhaps advertisers themselves) has argued that sponsored links provide any value to anyone but the sponsors. One such argument might be, a sponsored link increases the relevance score of the sponsor in a search engine for searches with terms related to the sponsor, and this is a good thing because the sponsor is a relevant result for such searches. A counterargument says paid placement is not a valid relevance metric, and cannot, by definition, improve the accuracy of a relevance score. As such, sponsored links do nothing but decrease the quality of search results. Since the links serve no actual advertising-to-eyeballs purpose, then this is their sole effect. Clearly, there are sponsors willing to pay for this effect.

A bunch of folks have suggested that it is not the content provider's responsibility to protect the quality of a search engine's results. But the question remains, is it wrong to accept money in exchange for performing an act that reduces the quality of something-- anything in the world, even a commercial product like Google search results ostensibly are-- if the act has no other positive effects?

And if we expect that search companies have developed their product to the point that the sponsored links have no negative effects on search results, is it wrong to accept money in exchange for sponsored links that have no such effect, if we know the effect was the intent of the sponsor?

Elsewhere in this thread, an employee of a major search company has admitted that the sponsored links on the oreillynet.com sites in question are known to have no effect on their search results. Unless the sponsors do not believe this assertion (maybe it's all a disinformation campaign!), I would expect that this admission would drastically reduce the value of these links to the sponsors. I'm surprised the links are still up. Right now, they're just a very poorly designed web ad. (I could be wrong, maybe they're getting enough clickthrus to justify the cost, but I'd bet they'd do better with a real ad.)

  Tim O'Reilly [08.31.05 08:09 AM]

Correction: I heard from the folks at Site-sift, and they pointed out that they do in fact have directory entries in many categories. I went back and looked at some of the pages they pointed out, and while many categories are empty, some do in fact have listings.

Similarly, when I looked again at the Web10 directory, there are sometimes listings below the fold.

However, the fact that there are so many empty categories, human readable data 'below the fold", as well as a disproportionate number of listings in the area of search engine optimization, suggests that these are not sites aimed solely (or even primarily) at human readers.

There's clearly a whole SEO-based economy that I don't understand.

While I'm on the subject of corrections, I will add that Arlen Ritchie's comments on 3Genius are thoughtful and on point. I'll remind people that I did not claim that *any* of the sites in question are "spammers." I reported that other weblogs were claiming that they are and criticizing O'Reilly for supporting them, and reported on my investigation of what turns out to be a very complex issue. The fact that Nat independently blogged one of the 3Genius sites as a cool site indicates that they are doing some good work in the travel area.

And as the debate on this posting indicates, there is a *lot* of disagreement on the net about where the boundaries of legitimate search engine optimization might be, and the right of sites to sell ad links.

I'm going to continue to look into and think about the issue, and will report back as I learn more.

  Kevin Morris [08.31.05 01:01 PM]

This is a great discussion, but I think it misses a couple of key issues. One of them is something I've been thinking a lot about lately and that is why I'm posting here.

Basically, the size and influence of google contributes to 'increasing returns to scale' on the web. What I mean by that is that without what is referred to here as 'spamming', in the natural progression of things, websites that are bigger have better search results. That's because they have had more time to gather links and gather "citations". If the site is well designed with SEO in mind they also have more internal links which boost their PR. Such sites also have the unique opportunity to create other big sites (more cheaply and more quickly) by linking to them from their first big site. So once you have one, it is *much* easier to get a second.

But the size of Google also means that new sites that enter into a crowded field will have a big task ahead of them to get noticed in google.

Google's PageRank is based on citation theory and in the original google prototype founders Page and Brin discuss this theory:

Intuitively, pages that are well cited from many places around the web are worth looking at. Also, pages that have perhaps only one citation from something like the Yahoo! homepage are also generally worth looking at. If a page was not high quality, or was a broken link, it is quite likely that Yahoo's homepage would not link to it. PageRank handles both these cases and everything in between by recursively propagating weights through the link structure of the web.

In section "2.1.2 Intuitive Justification", the paper also describes how a damping factor can be applied to single sites or groups of sites "and can make it nearly impossible to deliberately mislead the system in order to get a higher ranking"

...easier said than done. But I guess google has a lot of people working on it and that is probably the jist of the post made by Mr. Cutt from google...

So it's no surprise that new sites try to find some shortcuts to get noticed. Those shortcuts include purchasing text links. I don't think it's so bad. The site who 'hoards their PR', as was so appropriately described above, is going to have higher PR than the site that dilutes it by selling offsite links for monetary gain. Similarly, in the physical world, huge media giants sometimes dedicate valuable print space to articles and stories that help out small companies with a positive piece of press, even when that small company isn't big enough to justify it. But nobody questions that. In fact I think it is considered ethical and of "human interest". The difference is that on the web we would like to believe that every site should have an equal chance in the eyes of google. In practice that seems optimistic. In any event, the websites who don't have a PR7+ to dole out may feel that it is unfair that the owners of the ones that do can.

My pet peeve is sites that unscrupulously trade hundreds or even thousands of links with other willing sites who have reciprocal links pages. This is what really gives irrelevant search results because such sites seem to be of lower quality in general and are perhaps more likely to have nothing of value to offer.

In any event, Tim, I think you have taken the right approach. Kill off the misleading ads and then keep an eye on the whole thing. But if you look at the quote above, O'Reilly is much more like Yahoo than the vast majority of sites on the web and as such it might wish to use the same discretion in what goes on its website as they do in what goes in its (great!) books. But even if you do nothing different, Google created this beast and they are continually working on taming it, and in the end it is their job. Nonetheless, I hope Google will put more effort into penalizing sites that use link farms :)

  sodora [09.10.05 03:13 PM]

Our meta search engine was blacklisted by Google with no explanation. (www.sodora.com) The engine lists recent queries, and I can only assume that Google considers this 'spamming'. What ever happened to free search? Seems to me that Google should have a more proactive approach to link spamming...

  Uncover China [10.19.05 10:17 PM]

I've always wondered if for example a competitor sends your site to link farms and then later you get blacklisted, is there any way to appeal to Google?

  John Morris Fenley [10.31.05 09:39 PM]

Site Title:
My Highway of Life Had Many Detours
Site URL:
Site Description:
My life defined by stories covering many routes and locations, mainly in the USA and Africa, and the lessons learned.

  Mark Stiles Keller [12.11.05 11:30 PM]

Ever wonder how eBay gets top search results for its main page utilizing the back filler of its listings? Search Engine Marketing is turning into a whole new world with many colors in the rainbow rather than just white hat or black hat technologies. Add this killer tool to what you put into practice in your SEO efforts, for the ultimate compliment to your already thriving Seach Engine Optimization campaign: Ultra Ranker SEM Software Suite v3.0

  China E Guide [02.11.06 11:47 AM]

I think for links farms issue, if you appeal to google, they will not do a proper check until one year pass. It happened to me last time when i waited for a year.

  Thommes [04.21.06 05:43 AM]

I think there is a big different between sevices who just link automatically over software-tools with any sites, no matter which kind of content the site is about, and exchanges you can decide yourself to link with other sites.

  Greg [04.22.06 07:38 PM]

I have read this post with interest and as someone who reads and has purchased O'Reilly books on numerous occassions.

I would go so far as to say that my two main sources of trusted content online are W3c and O'Reilly.

For what its worth this is my opinion.

I think that content suppliers should be able to freely display advertising to anyone they feel meets there criteria. Once an advert is displayed this is then quite legitamately classed as a vote from that site to the advertiser. If the host does not like the advertiser site then they should not allow the link.

To display a link to an advertiser that is somehow discredited in code the user can't see ie.rel=nofollow tag but to not display this mistrust visually to the page viewer is a form of cloaking in my opinion. You are visually displaying a link and secretly in the code discrediting the link without displaying the change of status to the user of the web page.

From an accessibility point of view the link would not show the same change of emphasis on a screenreader. By using the rel=nofollow tag you are only talking to the search engines. The user is completely forgotten.

Either link or don't link is the answer. Websites should in my opinion be designed for users not search engines and search engines should not suggest code that causes a web user to see one thing and a search robot to see another.

I think web code is better left in the hands of "not for profit organisations" such as w3c. I am sure search engines such as google would be welcome to contribute to this forum.

  Greg [05.25.06 04:32 PM]

I would like too add something to my post. I have a 8 year old daughter thats very active on the internet and today I found links from a games site she goes on using the rel=nofollow tag to undesirable content. The site she was on was using the tag to exempt itself from penalisation from search engines but at the same time direct it's users (in this case children) to very undesirable content.

As I said before either Link or Don't link is in my opinion the only answer

  Mike [06.18.06 04:56 AM]

Search engine spam has become a major issue for google lately and more and more people (ab)using it, but I believe google are fighting it also, even though most of it brings them more money from adsense and adwords.

  James-A [11.17.06 06:40 AM]

Many people are using black hat seo technique to rank well in order to get their sites up in the Search engine rankings in a relatively short period of time. But how long are you going to fool google?

I think google should build some special tehniques to restrict this black hat seo guys.

  Ken [01.18.07 09:48 AM]

What's amazing to me when I came across this is that this was posted on Aug 2005 and "These ads have been running on O'Reilly Network sites for more than two years".

And to think that many people are just confronting this issue like it is a new problem or trying to do the exact same thing with search engines. Even better, there are sites that middleman the procuring and selling of text links.

So it seems like the situation is not geting any better.

  PatS [05.24.07 12:42 PM]

Link to it ...without a nofollow condom?
Great content begets, good and bad attention--been to a rock concert lately. As the engines get better, this is less and less of an issue.

I am a very happy O'Reilly user and always will be.

  Seamus [06.19.07 09:39 AM]

I can see the obvious benefit of the NOFOLLOW tag (to Google and other PPC providers), but it doesn't seem to have really gained much mainstream popularity, apart from on blogs(and other obvious spam targets).

Many SEO consultants who advocate the practice of NOFOLLOW tags and directives, purchase 'sponsored links' which don't appear to be of any benefit other than to their natural rankings.

  Mitch [07.10.07 07:22 PM]

You might want to read this first "Why Google Adwords is Not Helpful to Small Business" http://smartstartup.typepad.com/my_weblog/2007/07/a-fable-doing-b.html

  Oban [10.09.07 06:25 AM]

The reasons can be very confusing to non-tech people whether to implement a nofollow, and it's easy to understand their confusion. Personally I'd prefer not to have to use nofollow's as it's how the internet has started.

On the other hand I can also understand google's point of view in creating the best results for the user without the distortion of commercial interests.

A tag like advert or notrust etc could be invented to create category's of types of links with various weightings, but this would confuse people even more... :( So I think it's best to keep it simple...

  J.O. Urban [10.29.07 02:15 PM]

Funny how if everyone were to follow this procedure of slapping nofollow links on backlinks it would pretty much undermind the web and search results aswell. But in turn shift these advertising dollars to google systems (adsense/adwords). Just my thoughts...

Post A Comment:

 (please be patient, comments may take awhile to post)

Type the characters you see in the picture above.