How data and open government are transforming NYC

New York works to become a premier digital city.

New York City“In God We Trust,” tweeted New York City Mayor Mike Bloomberg this month. “Everyone else, bring data.”

Bloomberg, the billionaire founder of Bloomberg L.P., is now in his third term as mayor of the Big Apple. During his tenure, New York City has embraced a more data-driven approach to governing, even when the results of that data-driven transparency show a slump in city services.

This should be no surprise to anyone familiar with the mission statement of his financial data company:

Bloomberg started out with one core belief: that bringing transparency to capital markets through access to information could increase capital flows, produce economic growth and jobs, and significantly reduce the cost of doing business.

To reshape that mission statement for New York City, one might reasonably suggest that Bloomberg’s data-driven approach to government is founded upon that belief that bringing transparency to government through access to information could increase capital flows, produce economic growth and jobs, and significantly reduce the cost of the business of government.

As Gov 2.0 goes local, New York City has become the epicenter for many experiments in governance, from citizensourcing smarter government to participatory budgeting to embracing a broader future as a data platform.

One of the most prominent New Yorkers supporting architecting a city as a platform is the city’s first chief digital officer, Rachel Sterne.

Sterne gave a keynote speech at this year’s Strata NY conference that explained how data-driven innovation informs New York’s aim to be the nation’s premier digital city.

“I’m especially excited to be speaking with you because as a city, we need your help,” said Sterne to the assembled Strata attendees. “As the data practitioners and data scientists who are at the forefront of this revolution, all of our efforts are for naught if you are not part of them and not helping us to expand them and helping to really take advantage of all of the resources that the city of New York is trying put at your disposal.”

Video of Sterne’s talk is embedded below.

New York City’s digital strategy is focused on access to technology, open government, engagement and industry. “Industry is important because we need to make sure the private sector has all the supports it needs to grow and thrive and help to create these solutions that will help the government to ultimately better serve the public,” said Sterne. “Open government is important because if our data and our internal structure and priorities aren’t completely open, we’re not going to be able to enable increased [open] services, that kind of [open] exchange of information, etc. Engagement is crucial because we need to be constantly gathering feedback from the public, informing and serving. And access is the foundation because everyone needs access to these technologies.”

Big data in the Big Apple

What does data-driven innovation look like in New York City? Sterne focused on how data “evolves government,” asserting that it leads to a more efficient allocation of resources, a more effective execution, and a better response to the real-time needs of citizens. Although she allowed that, “as everyone knows, data can be manipulated.”

Sterne highlighted several data-driven initiatives across the city, including the Metropolitan Transit Authority’s Bus Time Initiative. “Initially, it was scoped out to hundreds of millions of dollars. The MTA ended up working with a local open-source development shop, [which] did it for a fraction of that, below a million dollars, and now you can get real-time updates on your phone based on where the buses are located using very low-cost technologies.”

New York City is also using data internally, explained Sterne — like applying predictive analytics to building code violations and housing data to try to understand where potential fire risks might exist. If that sounds familiar to Radar readers, it should: Chicago is also looking to use data, developers and citizens to become a smarter city. “This is as much about citizens talking to the infrastructure of the city as infrastructure talking to itself,” said Chicago CTO John Tolva in an interview last March. “It’s where urban informatics and smarter cities cross over to Gov 2.0.”

Web 2.0 Summit, being held October 17-19 in San Francisco, will examine “The Data Frame” — focusing on the impact of data in today’s networked economy.

Save $300 on registration with the code RADAR

New York City, however, has a vastly greater “digital reach” than Chicago. It’s bigger than many corporations and states, in fact, connecting to more than four million people through and social media channels that have expanded to include Twitter, Facebook, Tumblr, YouTube and Foursquare. Sterne envisions the city’s 200-plus social media platforms as a kind of “digital switchboard,” where citizens ask questions and government workers direct them to the appropriate resources, much in the same way that California connects citizens to e-services with social media.

The web as the 21st century public square

“What we’re really seeing that’s interesting about all these things is that they’re happening in public, so people are informing one another,” said Sterne. “They’re engaging one another, and it’s not so much the city telling you what to do but creating a forum for that conversation to take place.” If you visit NYC’s custom bitly URL shortener,, you can see what content is popular within that community.

Back in May, when NYC’s digital roadmap was released, Anil Dash highlighted something important: the roadmap captured New York City government thinking about the web as a public space. This has profound implications about how it should be regulated, treated or described. “The single biggest lesson I got from the
65-page, 11.8mb PDF is a simple one,” Dash, a native New Yorker, blogger and entrepreneur, wrote. “The greatest city in the world can take shared public spaces online as seriously as it takes its public spaces in the physical world.”

City as a platform

Sterne’s description of a “city as a platform” is one of the purest articulations of Tim O’Reilly’s “government as a platform” vision that I’ve heard any public servant articulate this year.

“The thing that’s really exciting to me, better than internal data, of course, is open data,” Sterne said during her Strata Conference talk. “This, I think, is where we really start to reach the potential of New York City becoming a platform like some of the bigger commercial platforms and open data platforms. How can New York City, with the enormous amount of data and resources we have, think of itself the same way Facebook has an API ecosystem or Twitter does? This can enable us to produce a more user-centric experience of government. It democratizes the exchange of information and services. If someone wants to do a better job than we are in communicating something, it’s all out there. It empowers citizens to collaboratively create solutions. It’s not just the consumption but the co-production of government services and democracy.”

Sterne highlighted the most important open data initiative that the city has pursued to date, the NYC DataMine. Soon, she said, they will be introducing “NYC Platform,” which she described as “the city’s API.” All of their work opening the data, however, “doesn’t matter if we’re not evangelizing it and making sure people are using it.”

NYC has used an app competition to draw more attention to its open data. As I’ve written elsewhere, by tying specific citizen needs to development, NYC Bigs Apps 3.0 is part of the next-generation of government apps competitions that incorporate sustainability, community, and civic value.

“We’ve had about 150 apps developed,” said Sterne. “There are apps that would be a significant cost to the city. Instead, they’re at basically no cost because the prize money is all donated. We provide 350 datasets. Until now, they were not API-enabled. They were not dynamic, but we’re going to be doing that because that’s the overwhelming response that we’re receiving from everyone.”

That feedback is widespread in the open government data community, where studies show that developers prefer to explore and interact with data online, as opposed to downloading datasets. When it comes to developers working with public data, dynamic access can open up entire new horizons for potential applications, as the release of real-time transit data has demonstrated.

Sterne shared some useful examples of apps that have been created using NYC open government data, including Roadify, which allows you to find parking spots or transit information, and Don’t Eat At, a Foursquare app that sends users a text message when they check into a NYC restaurant that is at risk of being closed for health code violations.

Sterne’s message to data scientists was generally quite well received at Strata. “Pleased to see @RachelSterne’s keynote today,” tweeted Alistair Coote, a NYC Web developer at RecordSetter. “If done right, open govt will be far more important than anything announced at #f8 today,” he observed, referring to Facebook’s new look.

Why open government data matters to New Yorkers

The experience in NYC during Hurricane Irene “once again proved the utility and importance of open data and the NYC DataMine, as several organizations used OEM’s Hurricane Evacuation Zone geographic data to build maps that served and informed the public,” Sterne told me via email. “This data has been public for over a year. Parties developing tools built on city platforms included WNYC, NYTimes, Google, Mobile Commons and Crisis Commons. NYC Digital was also in regular contact with these parties to alert them of information changes.”

The key insight coming out of that August weekend, with respect to the city acting as a platform during unprecedented demands for information, was that the open data that NYC provided on evacuation zones was used by other organizations to build maps. When buckled under heavy traffic, the city government turned to the Internet to share important resources. “As long as the right information was getting to citizens, that’s all that matters,” said Sterne at Strata. “It’s OK if it’s decentralized, as long as the reach is being expanded.”

As I reported here on Radar, the growth of an Internet of things is an important evolution. What we saw during Hurricane Irene is the increasing importance of an Internet of people, where citizens act as sensors during an emergency.

“Social media played a critical role in informing New Yorkers,” wrote Sterne. “Prior to that weekend, we established clear guidelines and a streamlined approvals process for social media content, which were disseminated to all social media managers. This ensured that even as we communicated in real time, we had accuracy and consistency in city messaging. @NYCMayorsOffice and were both major communication channels. @NYCMayorsOffice doubled its followers, increasing by nearly 30,000 during the weekend, and was cited by the mayor in press conferences as a resource. The YouTube channel was updated shortly after each press conference and saw nearly 60,000 views over the weekend. Over 32,000 tweets were published (not counting retweets) containing the text ‘nycmayorsoffice’ from August 25-29. Response was overwhelmingly positive.”

Data pitfalls and potential

Legitimate questions have been raised about New York’s data-driven policy, where journalists have questioned crime data behind the city’s CompStat program. The city has also faced challenges and nearly $300 million in expanded costs for its computer system for personnel data, offering up a sobering reminder of how difficult it is even for immensely successful private sector leaders to upgrade public sector IT systems. That’s a reality that former U.S. CIO Vivek Kundra can certainly speak to at the federal level.

That said, New York City and its mayor clearly deserve credit for opening data, being transparent about the administration’s performance, and continuing to work toward the incremental improvements that tend to be the way that government moves ahead.

For more insight into the IT behind New York City government, Radar’s managing editor, Mac Slocum, talked with Carole Post, commissioner of NYC’s Department of Information Technology and Telecommunications, about what being a data-driven city means and some of the most valuable ways that data is being applied. Video from that interview is below:

The challenges you see in opening up data in New York City are two-fold, and ones you see across government, said Post. “We first and foremost are a steward of the data that we hold, and so the concerns around privacy, confidentiality and public safety are definitely ones that need to be balanced against accessibility to the information,” she said.

That challenge is one that every big city CIO will face in the years ahead, as technology affords new potential to open government and new risks for exposing sensitive personal data. “While we are enormous proponents of having open data,” said Post, preserving integrity of the data and protections is important.

Post acknowledged that city government has “typically not been a very open bastion of sharing all of its information,” but pointed to a necessary step in open government’s evolution: moving to a standard of open by default, where civic data is considered open unless there is a reason for it not to be.


tags: , , , , ,