The Three Filters: How to Listen to Users

I really enjoyed Ken Norton’s essay User
Triangulation: When To Listen To Users
. It rang true for me,
as I’m spending my evenings wading through a pile of OSCON speaker
evaluations. It’s astonishing how often a single speaker gets both
“you sucked” and “you rocked” for the same talk. Mark Jason Dominus’s
Presentation Judo
addressed this: “Every
possible complaint will appear” … “You cannot win, so do the talk
in a way that preserves your professional dignity and integrity”
But conference evaluations are not the only reason Ken’s essay rang
true. It also cuts to the heart of what we do at O’Reilly.

When I joined, people often thought of us as a book publisher.
Now they think of us as a book publisher and conference organizer.
But that’s not what we do. We find people on the bleeding edge of
technology and report back to everyone else on what they’re doing
and how they do it. We believe that alpha hackers create tomorrow’s
mainstream technology.

In March I had a phone call with Liz Goodman of Intel. In
passing, while we were discussing Where 2.0, she said
that in what O’Reilly does there’s an implicit incorrect assumption
that what’s good for the alpha geek is good for the average user.
Alpha geeks have different problems, different knowledge, and
different skills than the average user. Selling software to the
mainstream based on the needs of the early adopter is an exercise
doomed from the get-go.

She had a great point–that is a large and unfounded leap
of faith. And yet I know that the people who watch the alpha geeks
see the technology before it hits mainstream. So there must be
something in the alpha geeks.

After a lot of observation of alpha geeks, observation of myself,
and observation of the dynamics of business, I figured it out.
Alpha geeks are technologists, so they are always the first to find
good technology. They’ll be the first to tell you when you’re using
bad technology (and here you need to be able to balance their
pronouncements against the tendency to favour the newest gizmo–today’s
iPod is tomorrow’s BetaMax). And they sometimes experience technology
problems that presage those of mainstream users–I had 20 spam
messages/day when my family had 2, I had 200 when they had 20, and
now we’re all using SpamAssassin (built by alpha geeks) we’ve all
returned to 20/day again.

But Liz was right, as Ken so eloquently expresses: alpha geeks
aren’t your users. If you want to know the problems your users
face, the way they use your product, and how well a given solution
works for them, you have to talk to them. You’ll have to conduct
research, listen to what they say, and critically evaluate to find
actionable signal in the noise. No matter how hard it is to figure
out what users need, you can’t learn it by speaking to alpha geeks.
But you won’t get hot technology tips from users, either. Know who
you’re talking to, weight what they say accordingly, and be sure
you’ve talked to everyone. You must filter and weight these inputs
according to who they came from and what they’re saying.

There’s nothing like taking a long spiritual
journey full of existential angst only to find someone you know
waiting at the other end with one eyebrow raised, asking “what took
you so long?”. Once I figured it out and talked to Tim, he pointed
out that he’s never talked about the alpha geeks as the be-all and
end-all for entrepreneurs. He’s always said that entrepreneurs
have to look through what the alpha geeks are doing to why they’re
doing it, and build from there. There’s a life lesson for me: if
you’re wondering what your job is, try asking your boss :-)

Because I’ve had my eye open for it, I’ve seen a lot of fellow
travellers on this road. For example, Marc Benioff (CEO of was part of an amazing back-and-forth at Supernova
with the room of alpha bloggers. The alpha bloggers asked him,
“why don’t you run a public-facing blog?” (he only posts on which requires
a Salesforce login) and Marc replied, “I don’t feel like blogging
to you. I want to talk to my customers and make them successful.”
It was fantastic! In a room of clique bloggers, Benioff pointed
out that they’re not the most important cog in his business. Listen
to it on IT
and enjoy (it starts at 16m30s). I was in the
room and wanted to whoop and holler. Marc’s mastered Filter
0: know you don’t have to listen to alpha bloggers.

Last week, Rael and I visited the headquarters of to meet with Mark Fletcher of
Bloglines and Jim Lanzone,
the SVP of Search Properties. Jim’s a very strong advocate of
considering what users need and do, rather than what simply what
alpha geeks say. We had a really strong discussion about the
difference between alpha geeks and everyday users. Jim pointed out
that Ask and Google are the only really pure search engines: people
may use AOL, MSN, or Yahoo! search because they’re already
AOL/MSN/Yahoo! portal users. As as result, Ask has to be heavily
focused on meeting user needs: if they give a crappy experience,
there’s no portal lockin (“we own your mail! use our search!”) to
keep them there.

As a result, Ask can’t follow the whims of the blogosphere.
They’re not buying companies or adding features just to be Web 2.0
or Ajax or whatever the alpha geek tech du jour is. They
only do things when there’s a tangible benefit for the end user.
Jim is a font of knowledge about what users do: 2% of users use
tabbed browsing, 13% of search is navigational (searching for
“” instead of typing it into the address-bar), users
look to the right side of the page to expand or contract their
search. I’m enjoying their “Zoom” feature on the right hand side
of search (check out the results of a search for search,
where an option to narrow your search is “directories”–a term that
isn’t textually related to the search term). Alpha-geeks have tried
this kind of search clustering before, but you have only to look
results for the same search
to see that blogger buzz doesn’t
equate to utility and value. Jim’s got a handle on Filter 1:
The best technology is necessary but not sufficient for success.

The lesson is that if you’re going to build a better mousetrap,
you have to approach the problem from both sides: technology and
user needs. If you ignore technology, you’ll build your mousetrap
from cheese. If you ignore users, you’ll catch people instead of

From the user side, I recommend Kathy Sierra’s Creating
Passionate Users
blog to keep you inspired and focused on the
user. Her tutorial was packed at ETech and at OSCON, and in a
recent post
she addresses the subject of fighting for users
from within a company. Her advice (“capture user stories”, “speak
for real users”) resonates with my experience organizing conferences.
My job with conferences is not budgeting, marketing, or sponsorship–it’s
purely to put together the best program I can, the one that will
meet attendees needs. To this end I read evaluations to find the
truth in the crossfire of opinion, I read blogs about my conferences,
and I talk talk talk to the users. I’m living Filter 2: a human
must make the final call between fact and crap

Nobody in their right mind builds a
business purely on technology. And nobody should guide their
business entirely by the words of alpha geeks. But as Yahoo!’s
stagnation in the first part of the new millennium (and their
resurgence once they rediscovered their geek roots) shows, you
ignore your trailblazer technologists at the peril of becoming a
Web n-1 company. And for O’Reilly, that’s why people meet
the alpha geeks at our conferences, read what the alpha geeks learned
in our books or on our
sites, and track what we’re
tracking via this blog.
It’s so they can get the data they need to filter to make a technology