How Long is Your City's Tail?

Here's something that everyone doing business on the web knows today, thanks to Chris Anderson: it's all about the long tail.

When the cost of each individual transaction falls to nearly zero, marginal and low-performing items, grouped together, can account for a lot more of the overall value of a company than the top-performing ones. Amazon.com makes more money from the aggregate of all of the books that sell one or two copies a month than from sales of best sellers. And Amazon is a much stronger, healthier, and richer company because of the extremely long tail of books it sells.

Everybody gets that.

What almost nobody realizes yet is that the same is true for cities - or can be.

Most cities right now are models of closed, rigid systems, systems that rely on a few, top-performing agents to get civic tasks done and keep quality of life high for residents. Most of these agents are departments of the city itself, though some are outsourced. Either way, cities rely on one agent per issue, no more. To use Amazon.com as an analogy, cities today are like an Amazon that only allows the #1 best-selling book from each category into its system.

A good number of cites are beginning to do deals with mega companies like Google and IBM, giving them access to city data so that they can build excellent tools for residents to use. This is a great thing. I love looking at Google Maps and seeing that bulging red line right next to my house indicating that traffic there is at a standstill so I should consider biking instead. Makes my life better.

Still, to use the Amazon analogy again, now these cities have allowed not just the #1 best-selling book from each category into the system, but best-seller #2, #3 and #4. It's still a closed system of cherry-picked agents with privileged access.

And that's about where a lot of people would choose to end the opening of cities' data - giving unrestricted access to the Googles, Microsofts, and IBMs of the world. The rest get limited access, or access contingent upon satisfying some governmental board or other. (New York's Mayor Bloomberg, who launched his Big Apps contest yesterday, is seemingly among this group.)

If we do that, of course, we're missing out on what is potentially the biggest piece of the pie - the tail. That's where a huge chunk of the value comes from.

So, imagine instead a city that has totally open, unrestricted access to data (say, San Francisco or DC in 2011). What does it look like? It has all of the familiar city-run departments providing all of the services and assistance they've always provided - that's not going away. Then it also has public services offered by the mega companies, the Google Traffic, IBM's Smarter Cities, and so forth. Those are huge added value to these open cities - they're used by a large percentage of residents and make life in those cities better. But THEN, it also has an insane long tail of services set up and run by anyone with an interest in doing so, just by hooking into city data, distributing it in a new way, improving on it, mashing it up, giving it back to the city, etc. These services each individually get used by a small minority of people, but collectively they get used by more than any other single source in the city.

That's the healthy, long tail city of the future in action: head, "meaty middle" and tail, all working together, all reinforcing each other, all driving each other forward.

And that's the future of cities.

So it might be time to ask yourself: how long is your city's tail shaping up to be? The answer may determine, to a large degree, how much your city is a thriving place to live in decades to come.

What Will Open Gov Look Like in Five Years and in One Year?

Someone asked me the other day what I thought open governance was going to look like in five years.

The more interesting question, I think, is what is it going to look like in a year?

Five years from now, the open gov ecosystem looks a lot like the web itself. It's huge. There are parts that are open source and used freely by all, there are parts that are proprietary and profit-generating. There's a healthy flow of information every which way - from government to citizens, from citizens back to government, from gov to gov, and from citizen to citizen. There are standards and best practices that are commonly observed. There is lots of movement from the bottom, up as well as from the top, down. There is high experimentation, rapid innovation, and rapid failure at low cost. There's an open marketplace for ideas, and good ones get rewarded with adoption.

Most importantly perhaps, the center of power in this ecosystem has shifted, from the offices of city halls or the White House outwards, resulting in a more balanced equation between governments and the people they represent. Government becomes less representative and more democratic. It also, by adopting web-like practices outlined above, becomes more efficient, leaner, less expensive to maintain.

That's the mature version of open gov. But what does the transitional version look like? How do we get from here to there? That's the really interesting question for someone working in this space. That's the burning question.

I'm sorry to report I don't have any hard answers for that question today, only questions of my own:

- Will businesses and entrepreneurs get into open gov and find it to be profitable and/or sustainable? Will there be a viable marketplace for open gov ideas and products?

- Will open source tools emerge that get adopted, used, and added to by different communities?

- Will there be a tsunami of cities opening up their data, one after the other, in a rush not to be the last open city out there, or will there be only a handful of open cities in a year's time?

- Will there be an attempt to standardize data structures across cities, states and federal offices, and will it gain any traction? Or will we be dealing with a tower of babel of local and open data?

All of these things remain to be seen, and they will affect what open governance looks like in a year's time. So without knowing these, one can only take best guesses as to what things will look like. And of course I have my own best guess, but I'll save that for another post.

I do know this however: a year from now, we'll see an open gov landscape that is very different from the current one of apps contests, dev camps, meetups, and open gov wikis. These things are the early-stage primordial ooze out of which the mature open gov ecosystem will grow. They are the organisms necessary for evolution to occur, but they are not the mature ecosystem itself.

It's time to begin to explore what the viable road from that early ecosystem to that mature ecosystem will be exactly.

Open Gov Is a Dialogue, Not a Monologue

At last week's Personal Democracy Forum I had a conversation with someone working for a city (I won't say which city), who was tasked with opening up that city's data. We were talking about the Apps for Democracy contests held recently in Washington D.C., and he explained his feeling about them:

"There were some interesting apps in there, but overall they didn't meet with the mayor's agenda for the city."

Being the non-confrontational person I generally am when in conversation with total strangers, I said "Oh yeah?" and the discussion continued without incident. Inwardly though I was thinking, did he really just say that? My god, this guy is missing the point ENTIRELY.

A city that opens up its data but expects that people building on that data should follow the mayor's agenda is going to fail miserably in its attempt at creating an open system.

Open government is about government as platform. And being a platform means letting people do whatever they like with your tools, letting them build in ways that meet their own agendas, not yours. It's about coming to see your users' agendas as your own agenda. If your users win, you win.

On the other hand, if you force your own agenda on your users, then they don't build anything, and everyone loses.

Open gov is a dialogue between governments and constituents, not a monologue. Everyone gets to decide what gets talked about and what gets built, not just the people with the data.

Everyone who works in the web understands this, of course. I know many of the people who are working on opening up government from the inside get this as well. But this conversation, with a senior-level employee at a government agency, made me wonder how many in government don't understand what open gov means, and what the real value and opportunity is to them. How many think of opening up APIs and such as a way to extend their own reach and increase their office's power?

This is one problem with grafting new ideas about platforms and APIs onto an age-old system rooted in a culture of contracts and RFPs. Can this graft produce a living, thriving hybrid of the two? Or will one necessarily become the subordinate of the other? If the latter is the case, which notion wins out? Open platform or fixed agenda?

Naming an Emerging Movement

There's a movement going on around the world.

We don't have a name for it, though.

Gov2.0, e-gov, e-democracy, open gov--these are all names that get applied to what is happening. And they are great for describing a certain aspect of this movement, the aspect that actually deals with government.

What's really going on right now is much bigger than that. Open gov is a big part of the story, but not the whole story. On top of Open Gov, there are organizations like The Open Planning Project (TOPP), or Front Seat, or my own DIYcity, working alongside these open gov groups, trying to make the whole civic system work better. Or there is Robin Chase, CEO of GoLoco and founder of ZipCar, singlehandedly trying to reinvent the way transportation works in cities. Or there is the subway alert I just got in my inbox, courtesy of New York City's MTA, notifying me that the F train has delays due to mechanical problems. All of these entities--TOPP, Front Seat et al, plus the open gov groups--are interrelated, and together create a new, emerging ecosystem of information, user activity, and possibility. But that ecosystem doesn't fit neatly under the hood of "Gov 2.0" or any of the other "gov" labels.

Recalling my post last week about the four pillars of an open civic system, these "gov" names--e-gov, gov2.0, open gov--focus on the G2C aspect of what is going on, to the exclusion of the other aspects of this open civic system that is emerging.

And this new civic system should have a name, because it is a real ecosystem. It is also a movement, with more and more people focusing on it around the world every day. It is also increasingly becoming an industry.

So what do we call this new thing?

What do we say when we want to say to someone, "All of the stuff that is emerging right now in the civic space that helps communities operate better, both with and without direct or indirect involvement on the part of the government?"

I was talking with Micah Sifry, co-host of this week's Personal Democracy Forum, a while back, and he suggested the name "civic software" for the apps that come out of this space. Riffing off that, I have been talking about the "open civic system." Fred Wilson of Union Square Ventures, who recently posted on this movement, thinks that name is too long, but also thinks "civic software" doesn't quite do it justice.

So I thought I would open up a thread here on Radar for a discussion:

What should this new space be called?

Let me know what you think. All ideas are welcome...

The Four Pillars of an Open Civic System

Everyone is talking a lot about open government and transparency these days. It's exhilarating stuff, and it's even more exciting to see governments get behind it, creating sites like data.gov in the U.S. for the public to access government information via APIs. But every time I hear someone say something like "our organization is really into transparency" (which is often) it sounds odd to me. It's only talking about a part of the puzzle, not the whole puzzle. What we really want (or what I really want anyway) is not simply government transparency, but an open civic system - a civic system that operates, and flourishes, as a fully open system, for whatever level we happen to be talking about - federal, state, city, neighborhood, whatever. And transparency is a big part of that open civic system, but it is still only one part.

In fact there are four parts to a functioning open civic system. These are:

Government to Citizen (G2C). This is what people speak of when they talk about transparency and open government data. It's the idea of creating open pipelines for data directly from government and gov't agencies to whoever is interested in receiving it. G2C gets you accountability - watchdog groups suddenly have easy access to the paper trails for everything that is going on, etc. It also gets you things like transit schedules, minutes from meetings, and zoning data - things that can be built on by third parties to make the civic system work better. G2C is critical stuff, but without the other three components in place, we can't make the most of this open government data. What we need is not simply a pipe of open data, we need an ecosystem of open civic data, all interconnected, all flowing every which way. That's what the other three "pillars" of an open civic system gets us.

Citizen to Government (C2G). The counterpart to G2C. This is the idea of creating open pipelines from the people directly to the government - hopefully with someone listening on the other end. Adding C2G to G2C completes the circuit and makes open government APIs and such that much stronger - it takes what was a uni-directional data flow and turns it into a feedback loop of information, input and output. At the city level, C2G is taking shape right now in the form of Open 311 - a open API that anyone can build on that allows residents to create "problem tickets" for their city to address one way or another. Washington D.C. is currently launching an open 311 API, and I expect more cities will follow suit soon. Other examples of C2G include UK's FixMyStreet and SeeClickFix.com from New Haven, Connecticut, both sites with a huge amount of potential. There are a million different, nuanced ways C2G could be played out, at the local, state and federal levels.

Citizen to Citizen (C2C). Okay so now we have both open G2C and C2G data flows going, and that's great - huge amplification of civic activity, great realization of efficiency with regards to interaction between government and people. But there are all sorts of ways to improve civic life that don't really need to involve the government at all - what about those things? That's where Citizen to Citizen, or C2C, data flows come in. C2C is the citizens' brigade of data flow - it's the people doing it for themselves, whatever "it" happens to be. Clever Commute, in New Jersey, is one example of a great C2C data flow. Everyone who commutes by train into NYC subscribes to the Clever Commute feed, and then notifies each other of what the current delays are, and where, each morning. The system works better than anything New Jersey Transit has been able to pull together, and at a cost of essentially zero. This is the great thing about C2C - it is added value to the civic system at no additional cost to the system itself. The cost to operate C2C is passed on to those who are using it, and spread out amongst individuals, to the point where the costs become negligible. Instead of New Jersey Transit coming up with a system that knows how late each of its trains are at a cost of millions of tax dollars, the users of Clever Commute bear the cost of the system, and it costs pennies for each user to operate (the cost of sending a text message). C2C is a huge value-add on top of G2C and C2G, and as governments consider how to get increased services in these recessionary times, I expect C2C to be huge - once governments get used to the idea.

Government to Government (G2G). Lastly, the square is not complete without open Government-to-Government data flows. Entities within governments should have easy, open data exchange with each other, without having to issue a request, parse something out of a PDF, and so forth. The ability for, say, the NYC Department of Health to get data from the Los Angeles DoH in realtime, without having to talk to anyone or issue a request could be a huge asset. Or think of the efficiencies that could be gained if the NYC DOT were able to exchange realtime data with the NYPD. If these examples sound vague, it's because G2G is the "pillar" I know the least about, having never worked in a government agency. From what I've learned though, it seems to me that there could be a huge increase to civic utility with a little bit of thought about an open G2G system.

And of course you can blend these data flows and come up with hybrids all you like. DIYcity's SickCity, for example, is basically a C2C tool in its present, basic 1.0 incarnation - it detects instances of residents in your city saying they're sick, and passes that news on to other residents. But a more sophisticated version of the tool would also pass that information on directly to the Department of Health when relevant, and would also, optimally, accept data from the DoH to pass that back to residents. Suddenly it has gone from a simple C2C tool to a tool that is C2C, C2G and G2C. Now we're talking about interesting stuff. Each additional channel of data makes the system exponentially more valuable.

With all of these systems properly developed and engaged, our civic systems - local, regional, federal - should bloom and transform into the properly modern, Internet-age things they ought to be. This will translate to increases in efficiency, greater innovation and rate of change, better adaptability, and greater resilience, in addition to other advantages. To get there though, we've got to get beyond thinking simply in terms of transparency and government APIs.

Trying to Track Swine Flu Across Cities in Realtime

John Geraci is a guest blogger and heads up the DIY City movement. He will be speaking about DIY City at Where 2.0 in San Jose on 5/20.

Since early last friday, when I got a tip about swine flu in Mexico City from a health researcher, the team that does SickCity has been working to make the system something that can (or could) detect swine flu outbreaks in cities around the world.

It hasn't been easy.

SickCity is the "realtime disease detection for your city", created by people at DIYcity. The service, launched last month, works by monitoring Twitter for local mentions of various terms that mean "I'm getting sick" and plotting those to location. Up until Friday, SickCity seemed to work reasonably well for the very rough beta tool that it is. It showed incidences of people reporting they had flu, or chicken pox, or other illnesses, broken down by city. You could look at a graph of the past 30 days for your city and see days when mentions of certain diseases and symptoms were higher or when they were lower. You could sometimes see trends. No one claimed that SickCity was ready for prime time, but those working on it felt that there was a very worthwhile idea in it that with a bit of refinement would be of huge value to communities.

On Friday, all of that got turned upside down.

Going to SickCity's Mexico City page early in the day, I saw a sudden, several-hundred percent increase in mentions of flu. The problem was, not a single one of them was about actually having the flu - all were about the gigantic swine flu media event that was just beginning. Our disease detection tool had turned into a media event detection tool overnight.

Since then, we've been in a constant struggle to filter out the media effect from the data. The problem is, as the story grows and changes, the terms we have to filter for keep growing and changing. On Saturday we made a series of changes to the filters and search terms, and thought we were fine. By Sunday, those had become totally insufficient in the face of the growing Twitter storm surrounding swine flu (70 more results in the time it took me to write that sentence). We made more changes Sunday. Today, those additional filters seemed puny and insufficient. People are now calling swine flu "piggy flu", "pork flu", "bacon flu", "wine flu". They're talking about Obama having flu. They're talking about bird flu. The list of tweeting topics grows at an exponential rate. The topic of swine flu is incredibly viral.

So how do you get down below this huge cloud of noise, to the relatively tiny (but very important) signal down beneath? There are probably several thousand tweets happening right now about the idea of flu for every one that is about actually having the flu. The number of people actually coming down with flu right now in fact seems very low (let's hope it stays that way).

Tracking other terms related to flu seems more promising - the term "fever" seems like a good one to look for, and once you get rid of the tweets mentioning spring fever, cabin fever and Doctor Johnny Fever, you've got a pretty good data set to use. But how representative of the flu population is that term?

Maybe tracking actual flu tweets in this situation isn't really possible?

Still, the payoff in terms of value to communities and health organizations is huge if the developers can get something that can be demonstrated to work. As a public health researcher following SickCity told me, realtime outbreak detection is currently terrible at best. To improve on what's there, you just have to give people a reliable signal that *something* is happening in a city. You don't need to have exact numbers. You don't even need to know whether what's happening is actually flu, or food poisoning, or plague, really - the health officials can figure that out for themselves pretty quickly with all of the other tools at their disposal, once they know to be on the lookout. You just need to be able to reliably say "there is a sickness event happening right now in this city", and that's enough. You just need a canary in the coal mine.

So the developers behind SickCity, volunteers from DIYcity (mainly Paul Watson and Dan Greenblatt at this time, plus a few others) keep working on making it that. And right now they're working round the clock. (It's a public project - if you want to pitch in, by all means do so - you can get more info here.).

Even if SickCity fails to detect swine flu in cities around the world, it will have become a much more robust tool in the process of failing. If it doesn't succeed in catching this pandemic, maybe it will be better prepared to catch the next one?

The Future of Our Cities: Open, Crowdsourced, and Participatory

Guest blogger John Geraci has spent the last six years making life in cities better with the use of web technologies. His latest project, DIYcity.org, has web developers and urban planners all over the world teaming up to create open source tools for residents of cities everywhere. Prior to DIYcity, Geraci co-founded the hyperlocal news network Outside.in.

Back in January, the city of Los Angeles announced a gap of $433 million for their 2009 budget. Instead of just cutting services however, LA Mayor Antonio Villaraigosa took the unusual step of posting a survey online for residents of the city to fill out. For each category of city service, the survey asked residents, "what program would you reduce to help balance the budget?", followed by an itemized list of services they could choose from.

It was in one sense a remarkable sign of the new openness and desire for participation sweeping government all over the U.S.

In another sense though it begged a larger question: if you're going to involve city residents in these issues, why stop at asking people which services they would like to cut? Why not go a bit further and ask them for input on how to keep these services, while making them leaner, more efficient, and smarter? And why not then ask for their help in making those changes happen?

These are questions cities everywhere should be asking today, as they find themselves faced with the challenge of gigantic budget shortfalls brought on by the recession. The conversation about the future of our cities should involve the people living in those cities. But it should not be about which services to eliminate, it should be about how to reinvent these services as modern, efficient things, how to make them work at a fraction of their current cost, and, while we're at it, how to make them better than they are now.

Why? Because cities don't have the money to improve, or even sustain these services on their own. Because people have good ideas, often more innovative than the ones coming from the cities themselves. And because increasingly, people have the means to actually build and implement these services - not as centralized, closed, top-down systems we think of as public services today, but as distributed, participatory web-based systems built using data open to all.

(continue reading)

Recent Posts