Three Paradoxes of the Internet Age – Part Two

Individual perception of increased choice can occur while the overall choice pool is getting smaller

This gem from Whimsley makes the point – with extensive statistical modeling supporting the argument – that our algorithm-obsessed, long tail merchants are actually depleting the overall choice pool despite the fact that as individuals we may be experiencing a sense of more choice through recommendations engines…

Online merchants such as Amazon, iTunes and Netflix may stock more items than your local book, CD, or video store, but they are no friend to “niche culture”. Internet sharing mechanisms such as YouTube and Google PageRank, which distil the clicks of millions of people into recommendations, may also be promoting an online monoculture. Even word of mouth recommendations such as blogging links may exert a homogenizing pressure and lead to an online culture that is less democratic and less equitable, than offline culture.

In short, the long tail has gangrene at its extremity – the niche. More disarming is the conclusion that it isn’t just the output of our recommendation algorithms that is leading to what the author calls “monopoly populism”and the end of niche culture:

“The recommender “system” could be anything that tends to build on its own popularity, including word of mouth…Our online experiences are heavily correlated, and we end up with monopoly populism…A “niche”, remember, is a protected and hidden recess or cranny, not just another row in a big database. Ecological niches need protection from the surrounding harsh environment if they are to thrive. Simply putting lots of music into a single online iTunes store is no recipe for a broad, niche-friendly culture.

The network effects that so characterize Internet services are a positive feedback loop where the winners take all (or most). The issue isn’t what they bring to the table, it is what they are leaving behind.

here is a link to yesterday’s post: More access to information doesn’t bring people together, often it isolates us.

Tomorrow: The myth of personal empowerment takes root amidst a massive loss of personal control.

tags: , , , ,
  • bowerbird

    once again, you’ve glanced off the point, not hit it head-on.

    yes, it’s absolutely true that our instant communication means
    that fads and trends can now grab bigger audiences than ever.

    probably the best example of that would be the “balloon boy”.
    who, by the way, was quite badly named since, remember?,
    he wasn’t actually in the balloon at all. but, thanks to twitter,
    many people thought that he was, and followed the action from
    that particular perspective. i don’t know if they felt stupid after
    they learned that the boy was not aboard, but i sure would’ve.

    nonetheless, lots of people got caught up in a real-time drama.

    it’s also now the case that the crowd has become “self-aware”,
    and — exactly like the crowd at a football stadium who starts
    doing “the wave” — they’ll generate stuff to amuse themselves,
    just because they can. was ashton’s million stunt anything but?

    but again, this is all to be expected.

    meanwhile, the long tail online is _still_ easily accessible,
    a huge difference from what the situation was like before.

    and finally — on something which is a pet peeve of mine —
    the collaborative filtering system used by amazon is a sham.

    “people who bought this book also bought that other book.”

    what’s wrong with that picture? it doesn’t tell us whether or not
    the person who bought this book, or that other book, liked ’em.

    do you like every book that you buy? i certainly don’t. i don’t
    even read ever book i buy, and i don’t like every book i read.

    so amazon’s so-called “collaborative filtering” is based on the
    wrong variable. that’s because amazon doesn’t even _care_ if
    you _like_ a book or not, they just care if you _buy_ the thing.

    “people who fell for the marketing campaign for this book
    also fell for the marketing campaign for that other book, so
    you too can be like those stupid people and fall for both books.”

    amazon built collaborative filtering only a capitalist could love.

    moreover, any recommender system based on _popularity_ is
    fatally flawed, from my perspective, because it can only point
    to what the mass public likes. i’m not gonna denigrate that by
    calling it “lowest common denominator”, because i _respect_
    the artist who can capture the attention of the mass public…

    at the same time, however, that’s _not_ what we really need.

    it’s quite easy to find out what “most people” like, because
    all we have to do is ask a sufficiently large sample of them…

    what we really need, though, to make the long tail _work_,
    is a collaborative filtering system geared to each individual.

    (it shouldn’t be necessary to phrase it in that way, because
    the definition of “collaborative filtering” should include that,
    but we’ve gotta rethink the mess the capitalists handed us.)

    what we need is a system that examines your ratings on a
    like/dislike scale for content you have consumed, and then
    compares your ratings to those made by other people, and
    then finds other people who gave ratings similar to yours,
    and then uses their ratings, on content you have not seen,
    to recommend — or disrecommend — that content to you.

    that’s what a true _collaborative_filtering_ system would do.

    and as soon as we are smart enough to put into place such
    a system — in an open-source manner, with public data —
    we’ll start reaping rewards, and the long tail will be wagging.

    this is the very mechanism that will link artists with audience,
    and thus free artists from the distasteful task of “marketing”.
    once an artist has found the slightest sliver of an audience,
    that sliver will slowly but surely pull the rest of the audience.

    but until we build it, we’re at the mercy of the capitalists who
    wanna sell us whatever it is they have sitting on their shelves.


  • Bowerbird –
    I think you are making a utopian argument that with the right technical calibration (open, based on likes etc.) we will solve the problem – I am not so sanguine. As the early semioticians said, “every decoding is an encoding” meaning – every time we try to unravel something – we are re-encoding our current biases into it. Humans are messy and thus our alorithms will reflect that mess. You have a direct and recent experience with this concept – for when you open your comment with “once again, you’ve glanced off the point, not hit it head-on.” you were decoding my intent through your own bias and reaching the conclusion that I had missed the point you were predisposed to make. I think I was making a different point. Let me explain:
    The argument I was putting forward (as interesting speculation) above is that wherever systems display network effects (the more mass = the more value and lock-in a product or service has)they can tend to diminish the overall choices available. One may argue with the argument – but that is the core of it to me – not how to better tweak the human-generated algorithm.

  • Randy

    Admittedly, my comment will not address Mr. Ross’s point or bowerbird’s response. But the following sentence may be the best summation I’ve read of the all too frequent Internet commenters’ ethos:

    “…you were decoding my intent through your own bias and reaching the conclusion that I had missed the point you were predisposed to make.”

  • bowerbird

    > for when you open your comment with
    > “once again, you’ve glanced off the point, not hit it head-on.”
    > you were decoding my intent through your own bias
    > and reaching the conclusion that I had missed the point
    > you were predisposed to make.

    my “once again” was referring to the first post in your series.

    just as i argued yesterday, in reaction to that first post, that
    the tangent to your point was more interesting than your point,
    i made the same type of argument today.

    i got your point today. i just disagreed with it. not totally,
    because there was something of interest. but tangentially.

    just like yesterday. which means it happened “once again”.

    given this trend, i expect that it will happen again tomorrow,
    or whenever you discuss the third “paradox” in your series…

    now, if you want me to get even more specific about how
    we disagree, i would be happy to do that, sir. you say that
    the overall choice pool is getting smaller. i say poppycock.

    you say that our current algorithms are narrowing the pool.
    i agree with that, to a degree, but i say that’s because they
    are being badly engineered, and if they were to be improved,
    they would actually widen our choices to use the entire pool.

    i even went so far as to tell you exactly how the current tools
    were being improperly engineered, and then explained the fix.

    so i think i was being quite specific in my discussion with you.
    but you reply with gobbledygook about “encoding” and “bias”.

    you predict the end of niche culture. i say it hasn’t yet begun.
    you say “the winners take all (or most)”. i call that “yesterday”.

    but please, if you think i’ve misunderstood something, say so.
    because i believe in dialog.


  • It is hard to believe that “Internet sharing mechanisms may also be promoting an online monoculture”. It is simpler for people to form specialized groups than ever before. So at least the blogs and forums in your niche would have stuff that is well suited to your interests.

    Finally, if so much of the tail content is really that great, then you could benefit from promoting it. This is an opportunity for entrepreneurs to bring a useful service.

    Having a Long Tail of content is not a goal in itself. Serving individuals find content they enjoy is a goal.

  • Richard Sever

    “every decoding is an encoding”

    I am not sure this was early semioticians but rather David Lodge poking fun at them in his novel Small World…

  • “Ecological niches need protection from the surrounding harsh environment if they are to thrive.”

    This is just weird; I don’t agree.

    Using the ecological metaphor, specialized behaviour by species that rely on other species don’t need protection from the complex arrangement they are a part of – they only need the complex arrangement they are a part of to continue. The authors’ construct seems to be the product of a top-down pyramid model of the ecosystem rather than the James Lovelocks’ Gaia model I see. But back to the internet, the formation and nurturing of niches should Increase the more the internet grows and becomes more complex.

    On other points you raise:

    I find Youtubes’ Recommended for YOU must be talking to someone else.
    (But then I’m making it hard for them to know me by not signing in and having several accounts.

    Related link-trails that interest me tend to peter out quickly.

    On recommendations engines – if I’m following these then I’m probably surfing around looking for a starting point. I rarely follow them. In entertainment sessions I find myself using the TV or radio schedule to twig me to shows I will go and watch on the net.

    Broadcast media – in the lineal time line they run in – have aggregating systems that push titles to the front at certain times. Broadcast media schedules remind me of Movies and shows that contain threads in their stories that send me in one direction or another. In this way I am following an algorithm.

    At other times I’ll use the internet as a tool. I’m using my bookmark list and highlighting words, ideas or names to open new tabs. Tab browsing takes me where I’m going, not where an algorithm is pointing.

    I don’t think I use the internet any differently than anyone else, so the ‘monopoly populism’ the author describes might be a bad read of the data or a extrapolation that is flawed – or he needs the Spectacle of Fear to market his writing.

    At any rate it doesn’t ring true to me.

    Michael Holloway

  • Thanks for posting this and for the comments.

    Several comments argue that I’m oversimplifying. Yes, and deliberately – that’s what simple models of complex phenomena do. My goal was to be one step above the anecdotal evidence that is often trotted out in support of the internet==diversity stance, and not to provide a waterproof rebuttal. There are other forces, and we all resort to several forms of information gathering as we navigate real and virtual space. But let’s remember that the same kind of subtlety applies to analog technologies too – if you reject this argument, let’s not hear “bricks and mortar culture is driven by the New York Times besteller list and prime time TV” which I do hear, a lot.

    @bowerbird – I do believe that the only real way of measuring diversity is through demand, not supply. The observation that, once blogged, a piece of writing is “available to everyone” is not
    interesting. What’s interesting is the patterns of consumption.

    @michael and @julien – interesting that you both say “it’s hard to believe” and “it’s weird”. Whether I have it right or not so far as overall diversity goes – whether that’s even a meaningful phrase – the central argument is that our individual experiences are not a good guide to collective outcomes, so it’s odd to see you arguing that it doesn’t match your own experience.

    Finally, @michael, I was thinking of ecological niches like the Galapagos islands, where there are species that not seen anywhere else, and which would not survive too many encounters with species from elsewhere. The first commenter to my original post says this much better than I could.

    Thanks again.

  • bowerbird

    > @bowerbird – I do believe that the only real way of
    > measuring diversity is through demand, not supply.
    > The observation that, once blogged, a piece of
    > writing is “available to everyone” is not interesting.
    > What’s interesting is the patterns of consumption.


    i don’t see how that relates to anything that i said.

    at all.


  • Well I was referring to this sentence: “meanwhile, the long tail online is _still_ easily accessible,
    a huge difference from what the situation was like before”. But it is just a little piece of what you were saying, I agree.

  • bowerbird

    > Well I was referring to this sentence

    ok, i see where you were coming from.

    still, you say this:
    > I do believe that the only real way of
    > measuring diversity is through demand, not supply

    and i maintain that all those blogs that are being read
    at the expense of the paper newspapers means that
    the _demand_ for this _diversity_ could easily be measured.

    likewise, the record companies are struggling to get even
    gold records today, whereas a decade ago it was platinum.

    and i’ve gotta believe that the plethora of new bands that
    can be _heard_ now, thanks to the internet, whereas before,
    they were sentenced to silence, means that you could easily
    measure the _demand_ for this _diversity_ in this sphere too.

    the television broadcast networks have lost viewers to cable
    every year for several _decades_ now, surely another sign
    of the actual _demand_ that is measured on a weekly basis.

    it all seems very clear to me, and there is no “paradox” here.


  • Jim Stogdill

    The Internet is a preferential attachment engine for ideas. It does for intellectual activity what Wall Street’s trading networks did for money. It accelerates the process of meme production and dissemination of memes but in a world If ideas that is opportunity flattened, outcomes are anything but.

  • @bowerbird, I have a different take on your examples . Comparing mainstream newspapers to blogs is apples to oranges. A real comparison to blogs would include not only mainstream newspapers, but also local papers, ‘zines, and small-scale specialist publications of all kinds. So while blogs have displaced mainstream news readership, they have also displaced (maybe more so?) small publications – and I’ve not seen any convincing stats to show that our reading is more diverse as a result.

    Something of the same goes for music. It is true that today’s big hits are less hit-like than a decade or two ago, but I’ve not seen anything convincing to suggest that the Internet has exposed today’s audiences to a wider range of music than in the past. Niche music tastes were plentiful decades ago too.

  • What I don’t understand is how the Google effect fits into your take. Meaning it’s the interest in a subject that makes me google around and follow links to what once would have been ‘zines that I would never before have known existed -or if I did, I might have had to track down an address and send a check etc etc. And I question whether cultural niches need Galapagos-style sequestering to thrive — in fact, most cultural niches are so uninteresting to the general public that noone will bother to tramp all over their territory. Instead, from what I’ve experienced, by making it easy for interested parties to locate media and become a part of it easily, I’ve seen the internet expand niches and make them more vibrant.

    My thought is that the desire for difference is not created by any cultural arbiter (or algorhithm), but inherent in the consumer. If you’ve ever tried to get a suburban kid to wade through NAKED LUNCH, you know the few who get excited are the ones who were already looking around for it.


  • bowerbird

    > I’ve not seen any convincing stats to show that
    > our reading is more diverse as a result.

    well, i know for a fact that _my_ reading is more diverse,
    and i don’t need any “stats” to “convince” me of that…

    just the fact that i am reading _you_ is more than proof.


  • Touche!

    But the whole point of my article was that a wider personal experience can (not necessarily must) coexist with less overall diversity, if our wider reading becomes more correlated. We all read more widely than before, but our readings may overlap more.

  • The popularity of content in a group of humans follows a power law curve. [citation needed]

    The internet makes it easier to produce content, so there is more of it, and easier to connect, so the group is bigger.

    As a result, the popularity power law curve scales up. It gets longer and it gets taller at the same time. That’s what scaling up means. This should not really come as a surprise, and isn’t really a paradox.

  • metatim – on power laws not being so pervasive after all, see And even if content were power-law based, there is no reason to think that different sharing/distribution/promotion mechanisms would have the same exponent.