The Three Filters: How to Listen to Users

I really enjoyed Ken Norton's essay User Triangulation: When To Listen To Users. It rang true for me, as I'm spending my evenings wading through a pile of OSCON speaker evaluations. It's astonishing how often a single speaker gets both "you sucked" and "you rocked" for the same talk. Mark Jason Dominus's Conference Presentation Judo addressed this: "Every possible complaint will appear" ... "You cannot win, so do the talk in a way that preserves your professional dignity and integrity". But conference evaluations are not the only reason Ken's essay rang true. It also cuts to the heart of what we do at O'Reilly.

When I joined, people often thought of us as a book publisher. Now they think of us as a book publisher and conference organizer. But that's not what we do. We find people on the bleeding edge of technology and report back to everyone else on what they're doing and how they do it. We believe that alpha hackers create tomorrow's mainstream technology.

In March I had a phone call with Liz Goodman of Intel. In passing, while we were discussing Where 2.0, she said that in what O'Reilly does there's an implicit incorrect assumption that what's good for the alpha geek is good for the average user. Alpha geeks have different problems, different knowledge, and different skills than the average user. Selling software to the mainstream based on the needs of the early adopter is an exercise doomed from the get-go.

She had a great point--that is a large and unfounded leap of faith. And yet I know that the people who watch the alpha geeks see the technology before it hits mainstream. So there must be something in the alpha geeks.

After a lot of observation of alpha geeks, observation of myself, and observation of the dynamics of business, I figured it out. Alpha geeks are technologists, so they are always the first to find good technology. They'll be the first to tell you when you're using bad technology (and here you need to be able to balance their pronouncements against the tendency to favour the newest gizmo--today's iPod is tomorrow's BetaMax). And they sometimes experience technology problems that presage those of mainstream users--I had 20 spam messages/day when my family had 2, I had 200 when they had 20, and now we're all using SpamAssassin (built by alpha geeks) we've all returned to 20/day again.

But Liz was right, as Ken so eloquently expresses: alpha geeks aren't your users. If you want to know the problems your users face, the way they use your product, and how well a given solution works for them, you have to talk to them. You'll have to conduct research, listen to what they say, and critically evaluate to find actionable signal in the noise. No matter how hard it is to figure out what users need, you can't learn it by speaking to alpha geeks. But you won't get hot technology tips from users, either. Know who you're talking to, weight what they say accordingly, and be sure you've talked to everyone. You must filter and weight these inputs according to who they came from and what they're saying.

There's nothing like taking a long spiritual journey full of existential angst only to find someone you know waiting at the other end with one eyebrow raised, asking "what took you so long?". Once I figured it out and talked to Tim, he pointed out that he's never talked about the alpha geeks as the be-all and end-all for entrepreneurs. He's always said that entrepreneurs have to look through what the alpha geeks are doing to why they're doing it, and build from there. There's a life lesson for me: if you're wondering what your job is, try asking your boss :-)

Because I've had my eye open for it, I've seen a lot of fellow travellers on this road. For example, Marc Benioff (CEO of Salesforce.com) was part of an amazing back-and-forth at Supernova with the room of alpha bloggers. The alpha bloggers asked him, "why don't you run a public-facing blog?" (he only posts on crmsuccess.com which requires a Salesforce login) and Marc replied, "I don't feel like blogging to you. I want to talk to my customers and make them successful." It was fantastic! In a room of clique bloggers, Benioff pointed out that they're not the most important cog in his business. Listen to it on IT Conversations and enjoy (it starts at 16m30s). I was in the room and wanted to whoop and holler. Marc's mastered Filter 0: know you don't have to listen to alpha bloggers.

Last week, Rael and I visited the headquarters of Ask.com to meet with Mark Fletcher of Bloglines and Jim Lanzone, the SVP of Search Properties. Jim's a very strong advocate of considering what users need and do, rather than what simply what alpha geeks say. We had a really strong discussion about the difference between alpha geeks and everyday users. Jim pointed out that Ask and Google are the only really pure search engines: people may use AOL, MSN, or Yahoo! search because they're already AOL/MSN/Yahoo! portal users. As as result, Ask has to be heavily focused on meeting user needs: if they give a crappy experience, there's no portal lockin ("we own your mail! use our search!") to keep them there.

As a result, Ask can't follow the whims of the blogosphere. They're not buying companies or adding features just to be Web 2.0 or Ajax or whatever the alpha geek tech du jour is. They only do things when there's a tangible benefit for the end user. Jim is a font of knowledge about what users do: 2% of users use tabbed browsing, 13% of search is navigational (searching for "www.example.com" instead of typing it into the address-bar), users look to the right side of the page to expand or contract their search. I'm enjoying their "Zoom" feature on the right hand side of search (check out the results of a search for search, where an option to narrow your search is "directories"--a term that isn't textually related to the search term). Alpha-geeks have tried this kind of search clustering before, but you have only to look at clusty.com's results for the same search to see that blogger buzz doesn't equate to utility and value. Jim's got a handle on Filter 1: The best technology is necessary but not sufficient for success.

The lesson is that if you're going to build a better mousetrap, you have to approach the problem from both sides: technology and user needs. If you ignore technology, you'll build your mousetrap from cheese. If you ignore users, you'll catch people instead of mice.

From the user side, I recommend Kathy Sierra's Creating Passionate Users blog to keep you inspired and focused on the user. Her tutorial was packed at ETech and at OSCON, and in a recent post she addresses the subject of fighting for users from within a company. Her advice ("capture user stories", "speak for real users") resonates with my experience organizing conferences. My job with conferences is not budgeting, marketing, or sponsorship--it's purely to put together the best program I can, the one that will meet attendees needs. To this end I read evaluations to find the truth in the crossfire of opinion, I read blogs about my conferences, and I talk talk talk to the users. I'm living Filter 2: a human must make the final call between fact and crap.

Nobody in their right mind builds a business purely on technology. And nobody should guide their business entirely by the words of alpha geeks. But as Yahoo!'s stagnation in the first part of the new millennium (and their resurgence once they rediscovered their geek roots) shows, you ignore your trailblazer technologists at the peril of becoming a Web n-1 company. And for O'Reilly, that's why people meet the alpha geeks at our conferences, read what the alpha geeks learned in our books or on our sites, and track what we're tracking via this blog. It's so they can get the data they need to filter to make a technology decision.