A Crowd-Sourced National Communications Census

My last tour of duty in DC was Chief Technology Officer at the Center
for American Progress. One of the fun things I got to do was figure out
what everybody else did, including my fellow Senior Fellows, the folks
that generated most of the policy work, many of whom are now occupying
senior posts in the new administration.

One of the most fascinating was Mark Lloyd. An experienced Emmy-winning television
producer, communications lawyer, and community activist, Mark is the
author of a well-regarded book about communications and democracy and numerous columns. He’s currently at the Leadership Conference for Civil
Rights.

The project Mark Lloyd was working on was a National Broadband Map to show our true
communications capabilities. And, he wanted to crowd-source the map from community
groups, supplementing that with census and other data from several different places to create a big mash-up. This was in 2005, around the same time Adrian Holovaty was thinking about chicagocrime.org.

I think the time is now ripe for this project, and when the new folks at the FCC asked me what I thought they should look at I pitched Mark’s idea (they’re reaching out to lots and lots of folks, which is a great sign). I asked for posting privileges here at Radar so I could pitch the idea to the Internet as well since I’m taking your name in vain as the folks that would make this happen.

The FCC has been charged to draft a National Broadband Plan next year. They’ve hired Blair Levin to run this process and he’s he started his new job in overdrive.
But, if we’re going to plan for the future, we have to know where we are today. When it comes to how we access communications in the United States, we really have no idea what we have and where it is.

The FCC collects broadband deployment statistics from industry. But, the
statistics are gross. If one site in a zip code has broadband, the whole area is considered high-speed. And, broadband is defined as 256 kbps in any one direction. Mark Lloyd is a long-time critic of these statistics: here are his 2004 columns on
Word
Games at the FCC
and
Catching
Up With Iceland.

Since we don’t have good statistics, our policy makers can’t make good policies. That
is apparent in two areas:

  • First, we don’t have an accurate ranking of our competitiveness compared to other
    countries. Each country measures broadband differently, and as you’ve seen we
    kind of cheat and say broadband is 256 kbps. Our policy makers have no idea
    how totally awful our communications capabilities are compared with,
    for example, Japan or Korea, where broadband starts at 100 mbps.
    What that means is that most of the international rankings we see in the
    papers from the ITU and OECD are just nonsense.
  • Second, we don’t have an accurate idea of the extent of the digital divide here in the U.S. Our
    inner cities, rural areas, and schools are all greatly under-served and our
    policy makers can’t see that when they administer programs such as the
    billion-dollar boondoggle that is the
    Universal Service Fund.

To the FCC’s credit, the new Chairman is well aware of these problems, and has commissioned Harvard’s Berkman Center to
review the existing data.
That’s a great start, but I don’t think you can just count on the industry and on consulting firms to get a good picture of our capabilities. I think you need hit the streets and collect real data.

To create a National Communications Census, the FCC could utilize two techniques:

  • First is basic survey science. Put a form up that allows people to enter their
    address, and answer questions: Is DSL available? Do you have it? Do you have
    cable Internet? Do you think it works well? Do you have cell coverage? What carrier? Do you have your own Wi-Fi? Can you access public Wi-Fi? In addition to a form on the FCC web site, an API would be great so people could develop their own apps for entering data from volunteer census workers on the streets.
  • Second is basic network research. Develop an API that allows an application to enter information such as Wi-Fi signal strength at a given location, cell carrier signal strength, and how many Wi-Fi networks are seen. Then, encourage people to develop iPhone, Android, and other applications to collect the data. Phones are actually great little limited-purpose spectrum analyzers. On your iPhone, for example, call *3001#12345#* and you can enter field test mode.

With both the survey and the data collection, the FCC can appeal to community groups to go out and war drive or poll-take in inner cities and rural areas so we can finally get a good picture of who has what. This is a lot of data that might come in, so rather than reinvent the wheel, it makes sense to partner with some non-profit network research operation like M-Lab which has the oomph to absorb an applications like this and can also make the data available
downstream to researchers, cartographers, and mashup artists.

Mapping our national communications capability is something the FCC needs to do for itself so it can create better plans, but also for state and local governments who are planning their infrastructure, and to the communications industry that needs to know
how and where to expand. A census would also of great use to ordinary consumers, who could use it see who has decent cell coverage, where to find Wi-Fi, or which Internet Service Provider works best in a given area.

tags: , , ,
  • http://baylink.pitas.com Baylink

    Quite a lot of the work is already done for you, here.

    There’s a version of the Kismet war-driving utility available now for Windows, in addition to the traditional Linux port and the module for WRT54G based routers running things like OpenWRT.

    This will do your survey for you, and knows how to do GPS too, and logs to a file format that people could upload to a concentrator node/system which could aggregate that data.

    On the broadband census side, talk to BroadBandReports: their speed test page already accumulates that data, I think; and that sounds like precisely the sort of project they’d be interested in.

    I believe they’re ad-supported; I’m sure they wouldn’t object to you driving them some traffic in the course of your census project. :-)

  • http://broadbandcensus.com Drew Clark

    Carl, thanks for your comment. One of the things that BroadbandCensus.com has been doing since our launch, in January 2008, is to provide a crowdsourced, public and transparent collection of data about local broadband Speeds, Prices, Availability, Reliability and Competition. We call this the Broadband ‘SPARC.’

    We also filed comments at the FCC in the National Broadband Strategy, which you can read at http://broadbandcensus.com/2009/06/broadbandcensuscom-urges-public-broadband-map-with-sparc-scores. We use the open-source Network Diagnostic Tool created by Internet2 for our tests.

    Mark Lloyd and I have talked quite a bit about the importance of this effort since at least 2006. That’s when I began an effort to make sure that the public had access to basic broadband data — or what we now call the Broadband SPARC — when I headed the Center for Public Integrity’s telecommunications project. You can read more about that effort here: http://arstechnica.com/tech-policy/news/2009/02/infrastructure-investment-decisions-need-transparency.ars.

    And as regards Baylink’s comments, I’ve reached out to BroadbandReports.com in an effort to find ways in which speed test information (those on BroadbandCensus.com, those by Measurement Lab, those on Virginia Tech’s eCorridors Program’s, and those of others like DSLReports/BroadbandReports), could all be mashed together and reused in a great variety of ways. The Berkman Center is also well aware of our efforts.

    At BroadbandCensus.com, everything on our site is published under a Creative Commons Attribution Noncommercial License, so that the data-sets we’ve accumulated can be publicly redisplayed and redeployed for free by academics, policy-makers and government officials.

    Additionally, on our news and events side, BroadbandCensus.com has provided news and information about broadband access, broadband policy and broadband deployment, as well as hosting the monthly Broadband Breakfast Club on the second Tuesday of each month. We publish timely and topical daily news on broadband, from the broadband stimulus package to proposals for a universal broadband fund; from the national broadband plan to wireless broadband offerings, as well as our subscription-based BroadbandCensus.com Weekly Report.

    Feel free to contact me at drew@broadbandcensus.com.

    yours,
    Drew Clark
    Executive Director and Editor, BroadbandCensus.com

  • Mike Pearson

    New Zealand already has a broadband map established. You can download the source code from the main site: http://broadbandmap.govt.nz/source-code/

  • http://dannyg.com/iapps/ Danny Goodman

    Re:Public form

    I don’t believe the average internet user could answer many of your suggested questions accurately. They most likely believe they’re getting more than they actually do, often thanks to advertising. AT&T has been promising DSL for my area at since before the Giants started playing at Pacific Bell Park. Comcast (our local cable monopoly) advertises the crap out of HDTV and super high speed broadband internet…but our entire area won’t get any of it for 2-3 years. Even so, I’d wager that most current Comcast customers around here who see those commercials believe all the goodies are available out of that little wire coming into their homes today.

  • http://theoephraim.com Theo Ephraim

    Partnering with broadbandcensus is great, but people that go to that site probably know and care about the issue and why we need to gather data. I think you should try to gather data from as many general purpose speed testing sites as possible. One very popular one is speedtest.net and I’m sure they have lots of useful data. By collecting from these sites you’ll get data from people who don’t know or care about the census as well.

  • Ricardo Suetonius

    A company called Skyhook Wireless in Boston already has a WiFi and Cell census to building level of the entire urban US. Most of Europe, Australia, Japan, and a host of other countries (or at least the urban parts of them) too. Perhaps someone should talk to them.

  • http://public.resource.org Carl Malamud

    The private efforts are all really good and should be recognized and encouraged. But, I do think it is time for the FCC and the NTIA to take a leadership role in gathering and publishing statistics. In particular, relying on voluntary industry-driven submissions when making policy is not enough.

    Just as the Bureau of Labor Statistics or the Census or other agencies are known as definitive sources and interpreters of data in their domains, I think the NTIA and FCC should take the lead in providing data on our national communications capabilities.

  • Ricardo Suetonius

    True enough, yet for anyone who wants or needs the most accurate roads data there’s no question but to go the private route – Navteq or TeleAtlas – rather than use the free, public TIGER option. And accuracy of the private stuff is aggressively driving Census/TIGER evolution, not the other way round. So maybe NTIA and FCC should grab that page from the Census playbook and tie into existing, evolving, private WiFI and cell signal inventories rather than set out to generate their own.

  • http://ellumobile.blogspot.com A Crowd-Sourced

    The project Mark Lloyd was working on was a National Broadband Map to show our true communications capabilities. And, he wanted to crowd-source the map from community groups, supplementing that with census and other data from several different places to create a big mash-up. This was in 2005, around the same time Adrian Holovaty was thinking about chicagocrime.org.

    I think the time is now ripe for this project, and when the new folks at the FCC asked me what I thought they should look at I pitched Mark’s idea (they’re reaching out to lots and lots of folks, which is a great sign). I asked for posting privileges here at Radar so I could pitch the idea to the Internet as well since I’m taking your name in vain as the folks that would make this happen.

    The FCC has been charged to draft a National Broadband Plan next year. They’ve hired Blair Levin to run this process and he’s he started his new job in overdrive. But, if we’re going to plan for the future, we have to know where we are today. When it comes to how we access communications in the United States, we really have no idea what we have and where it is.