Crowdsourcing and the challenge of payment

An unusual Distributed
Distributed Work Meetup
was held last night in four different
cities simultaneously, arranged through many hours of hard work by Lukas
and his colleagues at distributed work provider CrowdFlower.

With all the sharing of experiences and the on-the-spot analyses
taking place, I didn’t find an occasion to ask my most pressing
question, so I’ll put it here and ask my readers for comments:

How can you set up crowdsourcing where most people work for free but
some are paid, and present it to participants in a way that makes it
seem fair?

This situation arises all the time, with paid participants such as
application developers and community managers, but there’s a lot of
scary literature about “crowding out” and other dangers. One basic
challenge is choosing what work to reward monetarily. I can think of
several dividing lines, each with potential problems:

  • Pay for professional skills and ask for amateur contributions on a
    volunteer basis.

    The problem with that approach is that so-called amateurs are invading
    the turf of professionals all the time, and their deft ability to do
    so has been proven over and over at crowdsourcing sites such as InnoCentive for inventors and
    SpringLeap or 99 Designs for designers. Still,
    most people can understand the need to pay credentialed professionals
    such as lawyers and accountants.

  • Pay for extraordinary skill and accept more modest contributions on a
    volunteer basis.

    This principle usually reduces to the previous one, because there’s no
    bright line dividing the extraordinary from the ordinary. Companies
    adopting this strategy could be embarrassed when a volunteer turns in
    work whose quality matches the professional hires, and MySQL AB in
    particular was known for hiring such volunteers. But if it turns out
    that a large number of volunteers have professional skills, the whole
    principle comes into doubt.

  • Pay for tasks that aren’t fun.

    The problem is that it’s amazing what some people consider fun. On the
    other hand, at any particular moment when you need some input, you
    might be unable to find people who find it fun enough to do it for
    you. This principle still holds some water; for instance, I heard
    Linus Torvalds say that proprietary software was a reasonable solution
    for programming tasks that nobody would want to do for personal

  • Pay for critical tasks that need attention on an ongoing basis.

    This can justify paying people to monitor sites for spam and
    obscenity, keep computer servers from going down, etc. The problem
    with this is that no human being can be on call constantly. If you’re
    going to divide a task among multiple people, you’ll find that a
    healthy community tends to be more vigilant and responsive than
    designated individuals.

I think there are guidelines for mixing pay with volunteer work, and
I’d like to hear (without payment) ideas from the crowd.

Now I’ll talk a bit about the meetup.

Venue and setup

I just have to start with the Boston-area venue. I had come to many
events at the MIT Media Lab and had always entered Building E14 on the
southwest side. The Lab struck me as a musty, slightly undermaintained
littered with odd jetsam and parts of unfinished projects; a place you
could hardly find your way around but that almost dragged creativity
from you into the open. The Lab took up a new building in 2009 but to
my memory the impact is still similar–it’s inherent to the mission
and style of the researchers.

For the first time last night, I came to the building’s northeast
entrance, maintained by the MIT School of Architecture. It is Ariel to
the Media Lab’s Caliban: an airy set of spacious white-walled forums
sparsely occupied by chairs and occasional art displays. In a very
different way, this space also promotes open thoughts and broad

The ambitious agenda called for the four host cities (Boston, New
York, San Francisco, and Seattle) to share speakers over
videoconferencing equipment. Despite extensive preparation, we all had
audio, video, and connectivity problems at the last minute (in fact,
the Boston organizers crowdsourced the appeal for a laptop and I
surrendered mine for the video feed). Finally in Boston we
disconnected and had an old-fashioned presentation/discusser with an
expert speaker.

In regard to the MIT Media Lab and Architecture School, I think it’s
amusing to report that Foursquare didn’t recognize either one when I
asked for my current location. Instead, Foursquare offered a variety
of sites across the river, plus the nearby subway, the bike path, and
a few other oddities.

We were lucky to have Jeff Howe, the
WIRED contributor who invented the term Crowdsourcing and wrote a
popular book
on it. He is currently a Nieman Fellow at Harvard. His talk was wildly
informal (he took an urgent call from a baby sitter in the middle) but
full of interesting observations and audience interactions.

He asked us to promote his current big project with WIRED,
One Book, One Twitter
. His goal is to reproduce globally the
literacy projects carried out in many cities (one happens every year
in my town, Arlington, Mass.) where a push to get everyone to read a
book is accompanied by community activities and meetups. Through a
popular vote on WIRED, the book American Gods by
Neil Gaiman was chosen, and
people are tweeting away at #1b1t and related tags.


With the sponsorship by CrowdFlower, our evening focused on
crowdsourcing for money. We had a few interesting observations about
the differences between free Wikipedia-style participation and
work-for-pay, but what was most interesting is that basic human
processes like community-building go in both places.

Among Howe’s personal revelations was his encounter with the fear of
crowdsourcing. Everyone panics when they first see what crowdsourcing
is doing to his or her personal profession. For instance, when Howe
talked about the graphic design sites mentioned earlier, professional
designers descended on him in a frenzy. He played the sage, lecturing
them that the current system for outsourcing design excludes lots of
creative young talent, etc.

But even Howe, when approached by an outfit that is trying to
outsource professional writing, felt the sting of competition and
refused to help them. But he offered respect for Helium, which encourages self-chosen
authors to sign up and compete for freelance assignments.

Howe is covering citizen journalism, though, a subject that Dan
Gillmor wrote about in a book that O’Reilly published We the
, and that he continues to pursue at his Mediactive site and a new book.

Job protection can also play a role in opposition to crowdsourcing,
because it makes it easier for people around the world to work on
local projects. (Over half the workers on Mechanical Turk now
live in India. Biewald said one can’t trust what workers say on their
profiles; IP addresses reveal the truth.) But this doesn’t seem to
have attracted the attention of the xenophobes who oppose any support
for job creation in other countries, perhaps because it’s hard to get
riled up about “jobs” that have the granularity of a couple seconds.

Crowdsourcing is known to occur, as Howe put it, in “situations of
high social capital,” simply meaning that people care about each other
and want to earn each other’s favor. It’s often reinforced by explicit
rating systems, but even more powerful is the sense of sharing and
knowing that someone else is working alongside you. In a blog
I wrote a couple years ago, I noted that competition site TopCoder maintained a thriving
community among programmers who ultimately were competing with each

Similarly, the successful call center LiveOps provides forums for operators
to talk about their experiences and share tips. This has become not
just a source of crowdsourced help, and not even a way to boost morale
by building community, but an impetus for quality. Operators actually
discipline each other and urge each other to greater heights of
productivity. LiveOps pays its workers more per hour than outsourcing
calls to India normally costs to clients, yet LiveOps is successful
because of its reputation for high quality.

We asked why communities of paid workers tended to reinforce quality
rather than go in the other direction and band together to cheat the
system. I think the answer is obvious: workers know that if they are
successful at cheating, clients will stop using the system and it will
go away, along with their source of income.

Biewald also explained that CrowdFlower has found it fairly easy to
catch and chase away cheaters. It seeds its jobs with simple questions
to which it knows the right answers, and warns the worker right away
if the questions are answered incorrectly. After a couple alerts, the
bad worker usually drops out.

We had a brief discussion afterward about the potential dark side of
crowdsourcing, which law professor Jonathan Zittrain covered in a talk
called Minds
for Sale
. One of Zittrain’s complaints is that malicious actors
can disguise their evil goals behind seemingly innocuous tasks farmed
out to hundreds of unknowing volunteers. But someone who used to work
for’s Mechanical Turk said people are both smarter and more
ethical than they get credit for, and that participants on that
service quickly noted any task that looked unsavory and warned each
other away.

As the name Mechanical Turk (which of course had a completely
unrelated historical origin) suggests, many tasks parceled out by
crowdsourcing firms are fairly mechanical ones that we just haven’t
figured out how to fully mechanize yet: transcribing spoken words,
recognizing photos, etc. Biewald said that his firm still has a big
job persuading potential clients that they can trust key parts of the
company supply chain to anonymous, self-chosen workers. I think it may
be easier when the company realizes that a task is truly mechanical
and that they keep full control over the design of the project. But
crowdsourcing is moving up in the world fast; not only production but
control and choice are moving into the crowd.

Howe highlighted Fox News, which runs a UReport site for
volunteers. The stories on Fox News’ web site, according to Howe, are
not only written by volunteers but chosen through volunteer ratings,
somewhat in Slashdot style.

Musing on the sociological and economic implications of crowdsourcing,
as we did last night, can be exciting. Even though Mechanical Turk
doesn’t seem to be profitable, its clients capture many cost savings,
and other crowdsourcing firms have made headway in particular
fields. Howe hails crowdsourcing as the first form of production that
really reflects the strengths of the Internet, instead of trying to
“force old industrial-age crap” into an online format. But beyond the
philosophical rhetoric, crowdsourcing is an area where a lot of
companies are making money.

tags: , , , , , , ,
  • Bryce

    To try to answer your first question:

    1) No system will please everyone, and people will find cause to gripe about anything. So don’t set your expectations too high. Some volunteers will always feel underappreciated in such a system.

    2) It’s hard to answer the question in a way that covers all cases, so specifics would help.

    3) If you can set up well-defined tasks and quality thresholds for acceptance of the work, you could try an auction-style system. Tasks might start out as unpaid work, but be promoted to paid work as the need becomes critical, or if it’s been long-neglected (and therefore, we presume, “not fun”). To avoid duplication of effort, you’d probably have to allow participants to claim title to the right to collect on a task, for some reasonable period of time.

    The point is to make sure that the distinction between “paid work” and “volunteer work” is a distinction between the kinds of work, and not the status of the people doing it.

  • Dave Williams -Steward-

    What is the value of the crowd and their contribution to the organization?

    What would it cost to duplicate the efforts, and results traditionally?

  • Andy Oram

    Thanks for the two valuable preceding comments. Dave, I think your question is independent of mine, although someone may be able to tie them together. But your question is definitely a big part of the issues raised in the meetup and my blog.

    There are all kinds of studies trying to estimate how long it would take to develop the Linux kernel or some other key part of modern open source software if you paid developers standard wages. The amount easily tops a billion dollars. But of course, most Linux developers are paid these days. Certainly the value of Linux to its users hits many billions of dollars.

    Along those lines, your question is related to “What value to volunteers get from contributing?” Sometimes it’s direct value from using the results, and sometimes it’s less tangible satisfactions.

  • David Tanacea

    Instead of focusing on types of skill (professional vs. amateur or extraordinary vs. average) why not focus on asset delivery? Many crowdsourcing tasks create assets (web page, storyboard, software component, product design, etc.). Payment can be made based on the quality of the asset regardless of the perceived skill of the person creating the asset. This approach works as long as the mechanism for judging the quality of the asset is consistent and transparent. This process breeds consistency of quality since compensation is based on delivery of the work regardless of who is doing the delivering.

    Dave Tanacea, GM Enterprise Platforms, TopCoder

  • Kyle Hawke

    David T. put it best. Crowdsourcing will work best when compensation is based on merit – as determined for the most part by the crowd and to a lesser degree the client.

    But, compensation must be allocated on a sliding scale. The competitive, winner-take-all model used by some of the larger platforms is not sustainable. Eventually, after the intrinsic motivation wears off, those who are producing good work but are not (subjectively) selected to be the sole winner will go elsewhere. What was the crowd will again become the contractor.

  • Jon Payne

    Nice post Andy. I’m a big fan of 99Designs and oDesk. I think many business owners feel more comfortable when payment is made through a broker or service like this, rather than direct to the individual.

    I agree that merit-based pay is best, but agree with Kyle’s comment that sometimes it must be modified. With 99Designs, it tends to encourage volume of submissions rather than depth of submissions… lots of people take a stab but not many really spend the time to give it their best.

    Jon Payne
    Ephricon Web Marketing

  • bowerbird

    andy said:
    > How can you set up crowdsourcing
    > where most people work for free
    > but some are paid,
    > and present it to participants
    > in a way that makes it seem fair?

    you give yourself away in the question, andy.

    you only want the system to _”seem”_ fair…

    you don’t seem to care if it actually _is_ fair.

    the implication is you want to pull off a scam.

    re-work the assumption; you’ll find an answer.


  • Kiwi

    Kia ora Andy,

    May be you should have a look at
    They have sorted this challenge of payment about ten years ago. They also own


  • I agree with David T’s approach. It is more generic/universal and can be applied across multiple crowdsourcing platforms regardless of their particular business model.

    Kyle has a valid point, but I would argue that approached like 99designs are still sustainable as we are not talking about a static pool of providers where eventually the crowd becomes a “contractor”. There will always be an influx of challengers and their ability to gain reputation is only limited by their skills.

    Camilla Devreut
    Marketing Manager
    WebFrootz Web Agency

  • Its a hard problem to solve. And due to this dilema you are going to attract 3 kinds of entries.

    1) Those from developing countries (can be good).
    2) Those from hotshots who are so good that they win more than their share (good to have)
    3) Those that play the numbers game and try to enter many contests (not good due to low quality and plagiarism issues).

    One design crowdsourcing site that I saw recently offered contest holders the chance to offer a tip – that would be awarded to 2nd and 3rd place. Maybe even the top 4 or 5 entries should get a cut of the prize pool too.