How do we get government to share data?

A look at three approaches to opening government data streams.

On Tuesday we wrapped up the Manor Makeover in Manor, Texas, population 6,500. In some ways, this is ground zero for Gov 2.0 at the local level. The City of Manor has done some very innovative things on a shoestring, gaining attention ranging from the blogosphere to national press and all the way up to the White House. In fact, keynote speaker Beth Noveck, Deputy CTO in the Office of Science and Technology Policy at the White House wrote up a blog post just last night. The makeover is pretty impressive — they even followed my blog post from earlier this year about how to embrace Gov 2.0. (Not that they’ve seen it — it’s probably just obvious if you think hard enough about prioritizing limited resources.)

The champions of the Manor event (which “made over” a different city, De Leon, Texas) are Dustin Haisler (Manor CIO) and Luke Fretwell (GovFresh). They are both well ahead of the open government curve, and I think they’re absolutely on the mark: with a little elbow grease, the tools exist to make government more collaborative, transparent and participatory on a limited budget. With the right political backing, connections, knowledge and personal drive, a lot can be accomplished toward making government act more like a platform, as Tim O’Reilly and others have envisioned.

But how do we get this “perfect storm” of talent and resources to see it take off? There are more than 18,000 cities and counties in the United States. Are any others doing it this well? Actually, yes. But I can pretty much count them on one hand. Watching (and giving) some of the Manor presentations, I participated with the group in the promise of government as a platform. Bonded by our collective vision, we all bring something to the table: code, software, political/administrative expertise, or perhaps just a dose of unbridled enthusiasm. However, something is “rotten in the state of Denmark.” Just yesterday, Luke posted a blog entry announcing the “end of a GovFresh era” — and this, one day after its most successful event.

As I listened, my mind kept wandering back to one common showstopper: without laws to compel the sharing of local data, it’s never going to happen at scale the way people are hoping. Even if we can get past the biggest issues — inertia and resistance to change — municipalities still don’t have the budgets and technical know-how to make it happen. Oh I know, I know — there’s always the argument that “the ROI on giving out data is a no brainer”…sorry, but the evidence isn’t there yet. If it were so compelling, more governments would be doing it.

How, then, does the public get access to data, and ideally, to raw data streams? The typical hurdles to sharing information — technology, budgets, politics, and the absence of laws requiring its disclosure — are particularly acute in law enforcement, so it makes a good case study. Tight budgets are forcing many agencies to cut personnel — and budgets were already tight before the recession. Not to mention that crime data is at best controversial and at worst vulnerable to statistical manipulation by whomever can get their hands on it. So what can we do? As far as I can tell, there are three basic tactics:

  1. Using force, by changing the laws or creating new regulations requiring agencies to disclose data and provide it as a machine-readable feed. (Note: some sunshine laws give access to limited data, but not as an ongoing feed or through an API.) This will take forever, so it’s not a great option.
  2. Using intimidation, by enlisting the news media to pressure the agency, or by hiring lawyers to threaten them with lawsuits, which have no basis in law or fact (caveat: I am not a lawyer, but I have hired some to help me be more informed). Although some agencies might capitulate, most won’t. And then they’ll be put on the defensive.
  3. By creating value for agencies to entice them to share the data. Sometimes it’s as simple as asking; other times, they need to see how they will benefit or they won’t share anything. Still other times, it’s a non-starter.

Let’s talk about crime data

So what, exactly, is public data? Is crime data actually public data, and do agencies have to provide it? If so, why don’t they provide it as a raw data feed? The answer is this: while crime data concerns the general public, it is not exactly public data per se, and individual agencies have their own rules. Most won’t — or at least don’t — release it. And I don’t know of any agency, anywhere in the country of any size, that feels it is legally obligated to disclose any crime data to anyone as a machine-readable feed. Period, end of story. Please, prove me wrong.

So, given that #1 and #2 above are not going to be effective, and that there are no requirements to disclose data in an open format, let’s assume we have to use option #3 above, and create value to entice agencies to participate. We still have to deal with the barriers: budgets, technology and politics (fear, apprehension and skepticism about what will happen if they share it). While the barriers may not be completely justified, they are real.

The CrimeReports approach

At CrimeReports, there’s no question about it: we work for law enforcement. We create solutions for them. Because the laws don’t compel them to share information at all, they hold all the cards, so we draw them into voluntarily sharing crime data by showing them how it creates value for their agency. We solve their technology hurdles, and we make it affordable. Let’s face it, they have other things to worry about, like fighting crime.

One challenge on “making it affordable” is that the sweet spot price is just $100/month on average, and some still can’t afford it. No surprise that at that price, we lose money, and a lot of it, since it’s not easy to extract the data, standardize it and perform all of the work to make it available and easy to use. If it were easy, believe me, it would already be done.

The bottom line is that I think it’s good to have incident-level data available to the public in near real time. It’s good for communities. It’s good for the agency that provides it. And it’s good for individuals. But we need a business model that works. If we want that data to be out in any form, we need to have the ability to make up for the loss of our cost on performing that service.

To keep us in check, the invisible hand will exercise its magic. If we price it too high, someone else can move in and replace us, which has already happened (we actually have other viable competitors in the space). If you don’t think that will happen, think again. Our contracts are cancelable with 30 days notice, and we don’t have the exclusive ability to source the data. If you, your company, the Department of Justice, or the local newspaper wants the raw data, nothing is preventing that from happening. (In fact, some of our customers publish crime data to more than one entity.) You just have to make it worthwhile to the agency, since the other two options listed up above are unrealistic.

In a perfect world, every government agency in the country would provide an open and free data catalog. The early adopters have done it. More will continue to do so. We’ve come up with a process that works: it gets data out into the hands of the public, and that’s a step in the right direction. So let’s not let the perfect be the enemy of the good.

The discussion is far from over, and I think there’s more we can do. I’d love to hear what you think and keep the conversation running.

tags: , ,
  • Robert

    So what happens to the crime data itself once you normalize it? Do you share it or out it out in a feed? Please let me know! The creation of that data was paid for by public dollars …

  • Matt Rogers

    I found the crime data I believe you are referring to at I am not sure I understand how crime data as discussed here becomes public data simply by virtue of “it’s creation being paid for by public dollars”? Our dollars also paid for the all the fighter planes in the U.S. Air Force, but I still don’t have a single technical schematic for one of them.

    The fact is, the data is available and made so by law enforcement through several mapping services. I would rather my government spend my tax dollars to inform me, to help me protect myself, rather than rely on the resources of an already overtaxed system.

  • Robert

    I don’t think anyone is talking about security sensitive information here, Matt, so the fighter plane is a bit of a red herring. does not, as far as I can see, provide data even in a simple RSS feed as far as I can tell. you want to talk about openness: tell it to the hand.

    Maybe would respond well to “force” or “intimidation” or if someone created a better “value”, like text, email or RSS alerts, from this public, and not security sensitive, information.

  • Thanks for your comments! Individually:

    @Robert: Who said anything about normalizing the data? I think I smell a troll :D — But I can still answer: our agreement with agencies is as a vendor. We publish the data at a loss, so they agree to let us figure out other ways to make money with it. If we can’t, we shut down. If we raise the price, they don’t publish the data. That’s the whole point of my post.

    @Matt: Agreed, it’s the most cost effective and powerful way to get incident level data out to the public so they can make better decisions to protect themselves and their property.

    @Robert, revisited: Maybe would respond well to “force” or “intimidation” or if someone created a better “value”, like text, email or RSS alerts, from this public, and not security sensitive, information.
    That’s the beauty of our open, competitive market. If you can find a better way, go for it!

    Keep the comments coming!

  • Hi Greg,

    Good to read your thoughts on this topic. I think what you’re doing makes perfect sense from a running-a-company point of view. The fact that you don’t have some sort of long term country wide exclusive deal, leaves plenty of room for competition.

    Do I understand correctly that the $1 million “FY 2011 Commerce, Justice, Science and Related Agencies Appropriations Request” can be used to buy competitors services as well?

    The government might consider including a pass-on-the-raw-unprocessed-data clause or something like that in that budget.

    In fact, perhaps national legislation requiring government to include machine-readable-publishing in their contracts with data publishers might be an idea? That way you don’t force agencies to publish anything they don’t want to publish, but *if* they decide to publish it, it needs to be machine readable and reusable by the public. That’s a slow process, but they might be able to do it in specific instances like this $1 million budget.

    In The Netherlands (where I live) there are a few small experiments with crime mapping, mostly done by the police themselves. Police systems in our country are being unified as we speak, so these systems have the potential to serve the whole country.

    The problem with that unification is that it makes it more attractive for the police to build just one crime mapping site or award a 5 year exclusive contract to a single company (like we did with public transport data).

    I’m actually missing one tactic outside force, intimidation and creating value: create enthusiasm. Make people inside police organizations believe in the value of sharing data. Think of things that might happen and convice people to trust that these things will happen if they take a leap of faith and get their feet wet with open data.

    That’s what I’m trying to do and I was able to get one my of bosses to at least consider adding a phrase like “all publicly released information should also be available in a machine readable format” to a policy document. I also believe that the people currenlty working on a national IT strategy are open to these kind of ideas.

    The tricky bit is to find an example project that would lead to very short term results and require very little programming to build. I could use more ideas.

  • Reminds me of the stuff going on in Portland:

    @Sjors, you’ll find some good examples of using public data at CivicApps as well as

  • @Sjors: Hi, great to hear from you again. Responding to you:

    1. The appropriations request is just that: a request from a not-for-profit organization, not from us. I suspect that the group that requested it sees value in purchasing the service from a single entity (namely because of the network benefits.
    2. Your idea about legislation making data available in machine readable formats is great. Until some legislative body is willing to make that happen, we’ve come up with a great solution to make data available to everyone very inexpensively.
    3. I think we’re eye to eye on your last point, I am basically equating “creating value” with “creating enthusiasm”. Just saying here that until there are laws requiring the sharing of incident level crime data via a feed, we need to create enthusiasm among law enforcement or it won’t get shared.

    Please keep the ideas flowing. We really need it and thanks for your thoughts!