Everyblock's Code is Open-Sourced

The code for Adrian Holovaty's Everyblock has been released. The open-sourcing of the site's system were apart of the Knight News Challenge Program. Everyblock is a very impressive site that aggregates and geocodes local data -- news, crime, fire, restaraunt inspections and reviews - and then lets users define their interests down to the block-level.

Adrian made the announcement on 6/30. Here's the list of newly open-sourced, GPL'd goodies found on Everyblock's new Code page:

The main package (probably the thing you're looking for) is the publishing system, known as ebpub.
Second, the packages ebdata and ebgeo contain Python modules for processing data and making maps.
Third, the packages ebinternal and everyblock round out the code that powers EveryBlock.com. They're internal tools and are likely not of general use, but we're including them to be complete.
Finally, ebblog and ebwiki are our blog and wiki software, respectively. Because, dammit, the world needs another Django-powered blogging tool.

Django fans, Python geohackers and anyone who wants to build a local data aggregator are going to be thrilled. Adrian was one of the co-creators of Django and was one of the first Google Maps Mashup creators.

Everyblock has only launched in major US cities. There's plenty of room in the market for locals to create their own version. Everyblock spends a lot of time curating the incoming data feeds so I doubt that anyone will be able to roll out new sites too quickly. One thing to note: the trademark Everyblock is not available. However, the Everyblock team would not mind being acknowledged if you use their code. Personally I get a lot of value of Everyblock in my city. I get a daily email with all the crime, news and errata near my house.

Everyblock is now going to move onto the second stage of its existence. About five months Adrian blogged about the dilemna they would be facing when they open-sourced their software. As he said at the time:

But now we've reached an interesting point in our project's growth: our grant ends on June 30, and, under the terms of our grant, we're open-sourcing the EveryBlock publishing system so that anybody will be able to take the code to create similar sites. That's a Good Thing, in that EveryBlock's philosophies and tools will have the opportunity to spread around the world much faster than we could have done on our own, but it puts the six of us EveryBlockers in an odd spot. How do we sustain our project if our code is free to the world?

At the time I suggested that they try to federate with new everyblocks. After yesterday's announcement I mailed Adrian to ask him for a hint about their future plans, but for now he's keeping mum.

Bing's Sanaz Ahari on System Feedback (2 of 2)

A couple of weeks ago Bing had a small search summit for analysts, bloggers, SEO experts, entrepreneurs and advertisers. It was held in Bellevue; they put us up in the hotel and fed us. While there we received demos from Bing project teams. I was able to snag an interview with Sanaz Ahari, Lead PM on Bing. She led the team that developed the categories you see on a Bing web search. The interview was based on the slides from her presentation at the event. I have posted the significant images from her slides. The first portion of the interview focuses on how the Bing team handles Query level categorization and some of the problems they face. The second portion focuses on the systems used to generate the categorization.

Disclosure: I was on the MSN Search team (now the Bing team) from 2004- March, 2006. I knew Sanaz at that time.

Brady Forrest: Now on this image, it shows the ranking model and then it shows engagement and measurement.

Sanaz Ahari: Yes.

Brady Forrest: How does engagement and analytics factor into tweaking the ranking, measurement and engagement algorithms?

Sanaz Ahari: So the key thing about engagement is really there's two things: A, how often do people click on the different categories and then B, once they click on it, what do they do after that? So we basically feed that back into figuring out, "Okay. Did we actually put up the right thing? If something lower down is getting clicked on more, does it deserve to be higher? If something is not getting enough engagement, does it need to be bumped down?" And as we really expand the system, I'd have to say for us as a team, this is really the first step towards what we want to do. And, ideally, we want to get to the point for where we have enough understanding about every single query that we can really help you refine your tasks and your categories. So the engagement model can also help us in the future as we go in deeper into queries for helping people. We shouldn't just say, "Seattle, I'm going to Seattle restaurants." You should be able to go to Seattle restaurants and go in really deep and say, "I want restaurants in this neighborhood. I want of this price range, et cetera." So all of the engagement metrics can actually help us figure out what are the follow on tasks that users engage in the most as well.

Brady Forrest: And so what is the second flow chart?

Sanaz Ahari: So the second area, so once we felt that we could deliver intense understanding at a level of quality that we felt comfortable with, then we tackled the second area of problems which is equally difficult, which is really around, okay, how do we know that J Lo is a musician in the first place. And this is really around the query understanding aspect of things. And this is an area where we, again, explored multiple different approaches. We could've done a very kind of clustering on the entire corpus of our quarries. Or we could've said, "We're going to start a little bit more targeted and only go after the domains that we really want to go after." Like we said, "Let's just go after health and see if we can solve a small problem before trying to take on the entire corpus of the web."

For the Bing release, we focused -- and this was just like a principle that we had of the team was we really wanted to start small and see if we can get the level of quality that we wanted before trying to take on a lot more different challenges. And so, in this case, we definitely -- we went after the types of domains that we knew were strategic for us. So, all of the sudden, our corpus of quarries that we were interested in was a lot smaller. And we already have abilities to classify quarries into domains and understand, okay, this query is a music query or this query is health query, et cetera, et cetera. And so the other problems that fall out of that is, okay, when people do do health quarries, what are the categories that fall out of that? Like how do we know that people are going to care about diseases and symptoms, et cetera, et cetera. And then the next problem after that is how do we know that we have a comprehensive understanding of all diseases? So we may be able to understand that there are N different diseases, but how do we know that that’s actually a comprehensive list?

And then lastly, there's a problem about -- and this is one of the fascinating search problems -- is users query for the same thing in many, many different ways. So an example that I had was, for example, health is actually a very complicated one where the ALS disease is also known as Lou Gehrig's disease. And it's also known as one other thing which sounds kind of complicated. I don't even know how to say it. But there's lots of different ways that people basically query for the same thing. And so those were the three different problems that we really had to tackle in the query understanding space. So the two areas that we basically looked at was A, if we are able to identify a C set of quarries in a category, how can we actually really expand that out and be able to understand that we have a comprehensive list expander? Like if we start with N items, are we able to expand it out and get a more comprehensive list of items that are very similar to an existing C set that we started out with. And that's really the query expansion problem.


Brady Forrest: And what type of numbers are you talking about? Is it 100 or 1,000 or 100,000?

Sanaz Ahari: Oh, for the C sets? It completely varies. It completely varies. There are some categories that are small. There are some that are large. Like if you try to tackle musicians as a whole, that's huge. Whereas if you try to tackle like sports teams or something, that's pretty small. So it varies.

Brady Forrest: And are you pulling category names? Like are you pulling Wikipedia? Like proper nouns in the case of musicians or are you also pulling raw queries from the logs?

Sanaz Ahari: There's definitely both. We use a whole bunch of different features. We do a lot of work from logs. We do a lot of work on document extraction as well. What's very interesting is logs can give you a lot of great information where we have enough information. So it doesn't necessarily help you address the tail with precision. And document extraction can potentially help you with more comprehensiveness. And one of the things I would say is we also realize the good thing about approach on a whole actually, both on the intense extraction side and on query understanding side has been that it was an amazing learning experience for the team to tackle the problems one at a time because we realized there were so many intricacies that there are some things where we can build a generic system and it can help every category. But there were also cases where we would find a lot of intricacies in some categories where we had to do --

Brady Forrest: So what's a query that you're proud of that was like really hard and you feel like -- like an example of a query that really came a long way?

Sanaz Ahari: I actually don't have one at the tip of my -- I do like the experience for Jennifer Lopez because she has a lot of different attributes.

Brady Forrest: What's one that you really want to improve but you didn't want to tweak by hand?

Sanaz Ahari: Actually, the Jaguar one was one (Bing search), the one this morning that we talked about. That was a great query. And in some ways, I actually think we do a lot of positive things with that query. Like in one sense, I would say that we definitely deliver a diversified experience. And we at least capture the different intents. Whereas without the left rail altogether, you get the -- most users don't really go past the third algorithm result. And that in and of itself doesn't really give users enough diverse to creations [word] to say, "Okay. This is really my intent. And this is what I really want to dig down to." So on one hand, I like what we have done. But in the ideal scenario, I envision us being able to enumerate all of the different intents and all of the different tasks that actually fall under every single intent. So ideally, we should be able to call out animal, team, car, et cetera and then call out the individual tasks that the users want to do beneath every single one of them. There is -- the two areas that I really, really want us to improve is one, around that. I think that disambiguation is a pretty hard problem where we've barely scratched the surface. And then the second area is the depths of our coverage. You know, I really want us to have a much deeper experience where if I type in Indian restaurants in Fremont (Bing search), I should be able to still get a categorized experience where I can still dig in deeper.

Brady Forrest: What percentage of queries categorize the experience?

Sanaz Ahari: So today, 20 percent of our queries have a categorized experience. And the team is actively working on our next release where we are working on increasing both the quality and the coverage and specifically going more into longer queries.

Brady Forrest: Okay. Well, thank you very much, Sanaz.

Sanaz Ahari: Thank you.

The Web^2: it's Exponential, but is it Contracting or Expanding?

The theme for the Web 2.0 Summit this year is Web Squared. It is rooted in the idea that as the web morphs into less of a hub and spoke distribution model and more of a network of connected people and things, innovation and opportunity on it are growing exponentially. There has been a little bit of discussion on the Radar back channel about exactly what this means, or should mean, and Nat started things off with a thoughtful response that probably should be blogged as well. In particular he introduced feedback loops into the discussion, and with Nat's prodding, I decided to share my response to his email here. I've edited it a bit to make it a *bit* more cohesive, and while it isn't as structured as I would like, these are my thoughts about the exponential future of the web and a little bit about how that future might also impinge on the future of government...

I agree with Nat that feedback loops are a great mental filter to view the world. I read a little bit of Wiener and now I see feedback loops everywhere. Furthermore, what I like about them as a mental model, is that they help me understand the web at the ecosystem level rather than at the level of a specific technology. Wiener defined a cybernetic system the way engineers define a thermodynamic system. In thermodynamics, a system is closed if no energy crosses its boundary. A cybernetic system is closed when no messages or information cross. Since messages are the lifeblood of feedback these boundaries are important. As an example, open government stuff is so exciting to me because once computing systems connect between the web and government, the boundaries of previously isolated cybernetic systems (e.g. the people and its government) begin to be permeable. And once they are permeable to computing messages they will also be permeable to cultural signals that can create cultural feedback loops. That will cause state to change on both sides of the boundary. Two small isolated cybernetic social systems become one larger integrated one with new feedback loops in place.

Regarding the exponential theme, I'm not sure that innovation is progressing as an exponential over time - although, in fairness, I'm still working on my unabashed optimism credentials. But... In the 1920's automobile companies were springing up like crazy in America. It was the era before production methods became the dominant competitive weapon and anyone with a good idea for a better combustion chamber design or a valve train or a styling cue could still try their hand at building a car company. With access to tools, labor, and know how Detroit in the 20's was a very generative environment for automobile innovation. But by 1980 even DeLorean with a trunk full of coke couldn't afford the startup costs - a combination of more sophisticated design requirements and the changes in production scale economics made it impossible.

Are Data Centers the Economic Equivalent of Manufacturing Plants?

The interesting parallel with the web (or computing and software more generally) is that the rise of the data center as a key piece of competitive know how and, perhaps more importantly, capital cost. The question in my mind is whether utility computing enhances generativity, or by making it contingent on powerful interests, effectively stifles generativity in the long term despite the generative potential of the technology (I'm shamelessly borrowing the idea of contingent generativity from Jonathan Zittrain). And a related question, does the introduction of capital cost as a major factor in the eco system eventually make the web feel more like Detroit in 1980? Will it fundamentally change the web by tying it firmly to those who can access sufficient capital? (Google spent over $800m on data center capital improvements last year. That's a number that even makes the Defense Department wistfully declare "we just can't afford to do what Google does.").

Or, ...the electric utilities made innovation with electrical devices more possible, but it doesn't necessarily follow that utility computing will always do the same. After all, electrical utilities ship their power to us where we use it in situ for whatever purpose we want, but utility computing requires us to send our "loads" to them where it is much easier to implement perfect mechanisms of surveillance and enforcement. Homeowners associations used association charters to turn neighborhoods into little fascist fiefs and data centers have the potential to do the same with EULA's.

Scale and Concentration (or, is the Universe Expanding or Contracting?)

As scale on the web increases there are competing concentrating and generative factors at work (any of which might be exponential). The concentrating factors (need for capital, sophisticated expertise, ...) tend, like gravity, to collapse the system down on itself in a variety of ways. I don't mean that it becomes less relevant or makes less money, I mean that it ends up feeling more like AT&T in the 60's with centralized control and vested interests and strict contingencies on generativity. Just like Apple's oversight of the app store. On the other hand, the factors that tend toward expansion are feedback loops that span organizational boundaries, ready access to seed funding, standards for cloud computing that encourage true commodity availability of non-contingent generative environments etc.

Figuring out which force will dominate is like trying to figure out whether the universe will expand forever or eventually contract. The balance between the factors is quite subtle, depends on minute variations in initial conditions, and is very difficult to predict. But, we can still ask ourselves, "how can we influence the broader cybernetic ecosystem of the web to encourage policy, practices, cultural values, etc. that will promote generative expansion rather than scale-driven contraction?"

Exponential Effects and Social Structures

Shifting gears for just a moment, complexity science is the other idea I tend to come back to as a frame for viewing the web that, while not directly related to the exponential theme, is at least peripheral. The web is fascinating in the way it has become the cybernetic substrate on which both technical and social patterns are emerging. Stripes form on a zebra because "black" and "white" chemical messengers from adjoining cells interact with each other differently over distance. Out of that simple mechanism complex patterns emerge. The web is transport for human messages that don't decay with geo-spatial distance. This geo-and-time-independent messaging is enabling human "striping" that is no longer geo-ethnic dependent.

Within a geography the existing striping can become more severe as the web enables self-selected and self-reinforcing pockets of auto-propaganda that combine with social graph clusters; clusters that only infrequently span value systems. The situation is reminiscent of 1930's era Spanish political parties and their newspapers, but operating at photo-multiplier tube speed. We consume the stuff that reinforces our world view and segregate ourselves into more and more thoroughly strident neighborhoods of belief. We remain physically in our geo-defined country, but in our chosen echo chamber we each live a very different intellectual and emotional experience in a whirlpool of exponentially hardening world view. Perhaps someday we'll live in "nation states" that are stripes of psychographic and value alignment instead of stripes in geography.

Of course, it's true that as long as we are physical beings we will continue to stripe locally in our physical world. The cybernetic overlay in human relationships provided by the the web doesn't replace that reality, but by augmenting it and letting us stripe along lines of affinity and value system without regard to geography, it contributes to fissures in our geo loyalties. These fissures are important because States exist to govern the physical world (trade, law, taxes, defense...) but they depend on shared values and culture to function effectively. Just look at Iran today to see the effect of incongruent value systems on co-located peoples.

Sarah Milstein on Iranian Protests and Twitter

Interview with Sarah Milstein

In this 10 minute interview with Sarah Milstein, co-author the Twitter Book, she discusses how Twitter is being used by Iranian protesters and how Twitter has accidentally created a system not easily overwhelmed or controlled by authorities. She also talks about the continued evolution of Twitter over the past few months. I ask her to contrast the reaction to Twitter during the Swine Flu with the reaction to Twitter during the recent events in Iran, and it is clear from her answers that as Twitter becomes more familiar to the general public the significance and meaning of the platform are constantly evolving. Milstein comments on whether Twitter is becoming more "serious", and responds to the continued stream of stories by journalists who feel the need to pass judgment on this still-emerging communications platform. Milstein also discusses this week's 140 characters conference in New York.

On the Iranian protests, Milstein is very deliberate to say that the powerful aspect of Twitter during the Iranian protests is that Iranians within the country were able to use it to communicate with one another and with those outside of the country. Toward the end of the interview, I ask Milstein to comment on inadvertent transparency in the context of a previous post by Brady Forest. The Iranian protests story this week was as much about facilitating communications as it was about making sure that protesters were not communicating unintended information to the Iranian government.

(continue reading)

Four short links: 12 June 2009

1. New Media Challenges: Legal and Policy Considerations for Federal Use of Web 2.0 Technology (Center for American Progress) -- report on the issues around Web 2.0 use in Government, which include privacy, security, Public Records Act, advertising, etc. See also It's Not the Campaign Anymore: How the White House Is Using Web 2.0 Technology So Far from the same group.
2. Government Data and the Invisible Hand -- Ed Felten has written a fantastic piece on the relationship between data, presentations of the data, and the government departments that produce the data. It is full of powerful recommendations: The best way to ensure that the government allows private parties to compete on equal terms in the provision of government data is to require that federal websites themselves use the same open systems for accessing the underlying data as they make available to the public at large. (via timoreilly on Twitter)
3. Fast Modularity Community Structure Inference Algorithm -- This algorithm is being widely used in the community of complex network researchers, and was originally designed for the express purpose of analyzing the community structure of extremely large networks (i.e., hundreds of thousand or millions of vertices). The original version worked only with unweighted, undirected networks. I've recently posted a version that works on weighted, undirected networks. (via mattb on Delicious)
4. First Driver for USB 3.0 -- After a year-and-a-half's worth of work, Intel hacker Sarah Sharp announced that Linux will be the first operating system supporting USB 3.0. (via deusx on Delicious)

Four short links: 11 June 2009

Trends, Graffiti, Games, and Streaming Video

1. Trending Topics -- full source code for trendingtopics.org, Wikipedia trend analysis. Rails app running on the Cloudera Hadoop Distribution on EC2. (via mattb on Delicious)
2. Graffiti from Pompeii -- I can't help but read these as Tweets. Herculaneum (on the exterior wall of a house); 10619: Apollinaris, the doctor of the emperor Titus, defecated well here (see also olde style Twitter) (via OvidPerl on Twitter)
3. Online Games Dominate Beijing Startonomics -- presentations from sessions on Chinese game business at Startonomics conference. Though there are many differences between the US and China games market, the one that stands out most is China’s ability to massively monetize games. Tencent, a leading Chinese web portal, social network and game developer, famously announced revenue of over $1 billion earlier this year, much of it coming from their avatar service. (via TinaTranT on Twitter)
4. Ustream's Audience for Apple iPhone Announcement Greater Than Cable News -- Ustream is amazing, you can take a consumer handycam and video broadcast live to a greater audience than many TV shows get.

Twitter is Not a Conversational Platform

Perhaps the most common reason given for joining the microsharing site Twitter is "participating in the conversation" or some version of that. I myself am guilty of using this explanation. But is Twitter truly a conversational platform? Here I argue that the underlying mechanics of Twitter more closely resemble the knowledge co-creation seen in wikis than the dynamics seen with conversational tools like instant messaging and interactions within online social networks.

Wikis are causally thought of as platforms for "collaborative" document creation. But on Wikipedia, while many people share knowledge to co-create pages, the process is not formally collaborative in the sense that contributors are not cooperating with each other ways that form group identity (to paraphrase Clay Shirky from his book Here Comes Everybody). To the contrary, passionate experts write the majority of text, and a long tail of other contributors offer relatively few, small edits. Many users contribute nothing. Through this process, Wikipedia pages often become valuable repositories of knowledge.

Brian Solis recently posited the dichotomy of whether Twitter is a conversational or broadcast platform. New data bears on this. According to a Harvard Business School study, about 10% of Twitter users contribute roughly 90% of its content. Anecdotally, these 10% are subject-matter experts, passionates, mavens, and thought leaders who break news, write strong opinions, and tell jokes. Like on Wikipedia, most users merely read this information, and a modest number of people in the long tail use the information in the form of re-tweets, comments, corrections, and alternative opinions or links.

So while an individual user may use Twitter primarily as a conversational tool or a broadcast medium, in its totality, Twitter operates a lot like a wiki: as a knowledge-sharing, co-creation platform that produces content and allows its consumption. Conversation is perhaps the most simple and obvious form of collaboration, but would anyone claim that Wikipedia is a conversational platform? Despite the presence of information sharing, co-creation of an end product, and even discussion pages, Wikipedians on the whole aren't having conversations.

According to this argument, Twitter is no more a conversational platform than Wikipedia is. But is it a social networking platform? New HBS data showing that men have 15% more followers than women and being twice as likely to follow another man than a woman also bear on this to some extent. Authors Bill Heil and Mikolaj Piskorski state: "On a typical online social network, most of the activity is focused around women - men follow content produced by women they do and do not know, and women follow content produced by women they know. Generally, men receive comparatively little attention from other men or from women."

As in the case of the conversational platform, it seems that Twitter is also no more a social network than Wikipedia is. Wikis have user accounts and discussion pages, and it is possible for relationships to form. Twitter has user handles and direct messaging, and relationships can form. But social relationships on Wikipedia and Twitter are not a prerequisite for satisfaction and success (inasmuch as that can be defined). For instance, the popular and useful account @BreakingNews has hundreds of thousands of followers but participants in effectively zero engagement. There are many Twitter users who contribute large amounts of useful information and engage in relatively little conversation. And it is not common for people to describe Wikipedia as a social network.

Andrew McAfee notes that two useful Twitter traits are its asynchronous and asymmetric nature. These two traits are also critical to Wikipedia, but importantly much less so within popular social networking platforms like Facebook and MySpace. Thus, entities that are clearly social networking platforms can be but are not necessarily knowledge co-creation platforms, and entities that are clearly asynchronous knowledge co-creation platforms can be but are not necessarily social networks.

If microsharing tools resemble wikis more than conversational tools and social networks, this has huge implications for how people and organizations approach use of this emerging technology. Solis suggests, I think rightly, that "sometimes it's effective to...maintain a presence simply by reading, listening, and sharing relevant and timely information without having to directly respond to each and every tweet." The strategy of being a "lethally generous" member of a community would seem to be more worthwhile in this context, contrasted with the individual-level customer service approach of (for example) @ComcastCares.

This framework for thinking about microsharing platforms as knowledge co-creation enablers also puts Nielsen's recent data on Twitter's "user retention and loyalty" in a new light. When the average user is a consumer of the content produced by subject-matter experts and passionate mavens, how much does it matter if the majority of use is infrequent spectating (particularly when the information is archived for asynchronous retrieval)? As Shirky recently noted in his talk at the IAC/ACT Management of Change Conference that I attended in Norfolk, VA, such an imbalance of contribution is not a condition of failure for the platform or its users.

Finally, if microsharing is equated with knowledge co-creation, rules for attribution becomes an important consideration. But while the wiki attribution process has generally been worked out, attribution on Twitter is like the wild west - there are no rules; only conventions that are commonly accepted in some circles but not others. In addition, it is relatively easy to cheat the system, hard to catch someone doing it, and difficult to determine what the consequences are of such behavior. This problem will be a lasting one, requiring careful consideration by not only the user community, but also Twitter itself.

Hackers wanted! Scholarships available to coders who'll come to journalism and help save democracy

Guest blogger Brian Boyer is a hacker journalist who writes about the intersection of technology and journalism. He's worked at public-interest journalism site ProPublica and is now at the Chicago Tribune, building their new News Applications team.

It's not news that journalism is in crisis. CNN turned newspapers into first-day fishwrap and Craigslist killed the business model. Solutions are scarce, and our democracy is at risk. I don't have a chart to guide our way through the darkness to Citizenry 2.0, but there are some who can navigate the singularity.

Journalism needs great hackers. Not just nerds, but programmers who care -- about the values of journalism and the power of a free press to hold government accountable. Luckily, hackers are a freedom-minded bunch. The free software movement is rooted in many of the same principals that guide journalism. But news organizations aren't very sexy places to work -- especially now, as layoffs, bankruptcy and closures plague the industry. So how can we bring nerds to the news? One old-skool school is trying.

Free beer school!

Tell your programmer friends: The Medill School of Journalism at Northwestern University is giving away full scholarships, plus expenses, to software developers.They can get a masters degree in journalism, gratis, from one of the most prestigious J-schools around.

I recently graduated from the year-long program, during which I studied with with one other hacker and ~45 brilliant 'normal' journalism students. I interviewed lawmakers, farmers and shopkeepers and wrote stories about agriculture, waterways, and the diabetes epidemic in Illinois. It was difficult to shake my introverted, google-first, face-to-face-as-a-last-resort programmer nature. But it was also thrilling.

Journalism is an info-geek's dream. You're constantly learning new topics, speaking with experts, and distilling real-world issues to their essence -- all in the mission of informing the folks who don't have time to soak up all that data. It's like being paid to write a new Wikipedia article every day.

We also wrote some software. My programmer colleague and I banged out enviroVOTE in a frenetic weekend of coding and coffee in the days preceding the election. The night of, we were tied to our keyboards, tallying results and tweeting updates while the rest of the world was watching TV. Such is the life of a journalist.

For our final project at Medill, the two coders and four non-coder new-media students built NewsMixer, an experiment in integrating social networks with news coverage. It was one of the first applications to roll out on Facebook Connect, and remains one of the only apps that explores its full potential. All the code is GPL'ed and has already spawned other open-source projects.

This is the time to remake journalism

Programmers have been making an impact in the news world for some time, but until recently most innovation in this space has been in creating new ways to present the old style. With a few shining exceptions like the datavisuals by the New York Times, most online news could have been written on a typewriter and mailed to Google for indexing.

Then, something amazing happened: Software won a Pulitzer Prize. Created by hacker journalist Matt Waite and other fantastically clever folks at the St. Petersburgh Times, PolitiFact is form of news that could only exist online. Aron Pilhofer, leader of the innovations team at the NYT, put it perfectly:

But is it journalism, some people asked? There's no lead per se, no narrative and no pyramids anywhere to be found, much less the inverted sort.

Journalism is about helping people make sense of important issues, and how those issues affect them personally. It's about uncovering that which someone wants to keep hidden. It's about holding people we place in high public office accountable. And by those definitions... PolitiFact more than meets the test. It takes a traditional form of newspaper reporting -- fact-checking what politicians say -- and scales it up in a way only possible on the web.

The NYT's Represent and its open-source cousin, Repsheet, are innovations much in the same vein, and their existence is a sign of the times. The tools now available to hackers are so great that we can think far beyond content management systems. The moment has come when a couple of great hackers can knock out a fully-fledged new form of media in a matter of weeks. Tell the Twitterati: there are lights in the distance.

Hackers wanted

The news is waiting to be saved. We have the technology, all we need is more nerds. So ditch your boring corporate gigs and come to journalism! Democracy is one hell of a fun problem to hack.

Building Bridges with the U.S. Intelligence Community

Guest blogger Jeffrey Carr is a cyber intelligence expert, Principal of GreyLogic, columnist for Symantec's Security Focus, and author who specializes in the investigation of cyber attacks against governments and infrastructures by State and Non-State hackers. Jeff is the Principal Investigator for Project Grey Goose, an Open Source intelligence investigation into the Russian cyber attacks on Georgia in August, 2008.

About three weeks before the start of the Russia-Georgia war last August, the Office of the Director of National Intelligence issued a directive entitled “Analytic Outreach”. In it, DNI McConnell authorized members of the 16 agencies that comprise the U.S. Intelligence Community (IC) to reach out to people outside the IC, “to explore ideas and alternate perspectives, gain new insights, generate new knowledge, or obtain new information.”

As someone who writes about Intelligence and National Security matters, particularly in the area of Cyber Warfare, this Directive was pretty inspiring to me. I had long held the opinion that Web technologists and researchers had an important role to play in Government. Unfortunately, I had no way of communicating that vision to anyone who mattered so I just decided to act on my own and launched an Open Source Intelligence gathering effort called Project Grey Goose, which brought together an eclectic mix of hackers, spooks, and techies from inside and outside the Intelligence Community.

Imagine how happy I was six months later to hear about a formalized and much easier way to bring outside expertise into the IC thanks to the dedicated efforts of a few intelligence professionals and the Deputy Director of National Intelligence for Analysis. Appropriately enough, this project is named BRIDGE.

According to its creator, Dan Doney, BRIDGE hopes to do for Public-Private collaboration what the iPhone Apps Store has done for the iPhone and its customers--produce a mind-boggling explosion of innovative applications for use by the Intelligence Community. We aren't at the mind-boggling stage yet because BRIDGE is still in its infancy, but there are some pretty cool apps which I'll describe in a moment.

In addition to being a development sandbox, BRIDGE also allows intelligence analysts to interact with outside experts whether they be in industry, academia, or other government agencies at the Federal, State, Local or Tribal level. Alternative analysis has long been a recommended approach to avoid myopic thinking by specialists. BRIDGE provides a platform for debating alternative viewpoints and comparing evidence across agencies, specialties, and borders of all kinds.

(continue reading)

The Lean Startup Talk From Web 2.0 Expo

One of our most popular talks at the Web 2.0 Expo SF was Eric Ries' The Lean Startup: a Disciplined Approach to Imagining, Designing, and Building New Products. I've embedded an audio version of his slides above. Eric recommends the talk for people who want to:

• Identify a profitable business model faster and cheaper than your competitors.
• Continuously discover what customers want to buy before building or making follow-on investments in new features.
• Ship new software at a dizzying pace: multiple times a day while improving quality and lowering costs.
• Build a company-wide culture of decision-making based on real facts, not opinions.

Eric has a follow-up to his talk and more thoughts about Lean Startups on his blog.

Importance of Innovation in Finance & BarCampBank

“Progress is not the mere correction of evils. Progress is the constant replacing of the best there is with something still better.” -Edward Filene

Two years ago, when we were organizing the first BarCampBank in the US, many people found it hard to believe that banks & credit unions could a place for meaningful grassroots innovation. Even crazier was the idea of organizing an unconference to begin bringing open source, transparency, identity, and community into the very closed world of banking & finance.

Since then the BarCampBank idea has turned into a movement. There have been over 14 events all over the world, and many of the ideas generated are beginning to turn into action.

To me, the global financial system is a platform that exists to “create more value than it captures”. Tim explained this in his Work on Stuff that Matters post, saying:

“A bank that loans money to a small business sees that business grow, perhaps borrow more money, hire employees who make deposits and take out loans, and so on. The power of this cycle to lift people out of poverty has been demonstrated by microfinance institutions like the Grameen Bank. Grameen is clearly focused on creating more value than they capture; not so the like of Fannie Mae and Freddy Mac, or WaMu, or many of the other failed financial institutions involved in the current financial meltdown.”

There has never been a more important time to bring meaningful innovation into the financial system, and there has never been more opportunity for our community to make it happen.

The next event is occurring this weekend (April 25-26, 2009) on Treasure Island in San Francisco.

After that, the following events are planned:

• BarCampBankVegas is set for May 2, 2009.
• BarCampBankCharleston2 is set for June 13, 2009
• BarCampBankGermany is set for October 23-25, 2009

Four short links: 6 Apr 2009

Baby nerds, evil URL shorteners, reasoned discussion, and the Government straps its Web 2.0 on:

1. Books for Wee Nerds -- Forget Pat the Bunny -- your baby wants to Pat Schrodinger's Kitty! Help baby search for subatomic particles and explore the universe. (via Tim's tweets)
2. On URL Shorteners -- Joshua Schachter and Maciej Ceglowski on the downsides of URL shortening services like bit.ly et al.
3. Mending The Bitter Absence of Reasoned Technical Discussion (Alex Payne) -- We’ve come to accept that trying to have a reasonable discussion on the Internet is like insert any number of increasingly offensive metaphors here. Usenet, IRC, forums, blogs, and now media like Twitter have all been black-marked as houses unfit for reason to dwell within. And so we roll our eyes, sigh, and quietly accept the idiocy, the opportunism, and the utter disrespect for our peers and ourselves that is technical discussion on the Internet. This need not be the case. It is possible to have a reasoned technical discussion on the Internet. People do it every day, particularly in smaller online communities where social norms are easier to enforce. We can do it. (via SarahM
4. GSA signs agreements with Web 2.0 providers -- Flickr, YouTube, Vimeo, and blip.tv get agreements that make it legal for federal agencies to use those tools. Followup to my earlier cite of roadblocks to Web 2.0 tools for government use. (via Fiona's delicious links)

Recent Posts

• Web2Open: An Exciting Experiment | by Sarah Milstein on March 29, 2009
• Web 2.0 Expo Preview: Will Wright, Sims and Simulations | by Kurt Cagle on March 26, 2009
• Web2Open: Great Sessions, Recessionary Pricing | by Sarah Milstein on March 24, 2009
• Transforming the Relationship Between Citizens and Government: Making Content Findable Online | by Vanessa Fox on March 24, 2009
• Ignite SF @ Expo: 4/1 at Mezzanine | by Brady Forrest on March 23, 2009
• Request for ideas: Crowdsourcing the Evolution of Congressional Websites | by Mike Honda on March 19, 2009
• Wikirank: A Zeitgeist for Wikipedia | by Brady Forrest on March 19, 2009
• The Social Nervous System Has More Than One Sense | by Tim O'Reilly on March 12, 2009
• Two Thousand People Singing Daisy Bell Together via Mech Turk | by Brady Forrest on March 11, 2009
• Startup Marketing Isn't Rocket Science, So Don't Hire the Ph.D Too Soon | by Darren Barefoot on March 9, 2009
• Vivek Kundra: Federal CIO in His Own Words | by Timothy M. O'Brien on March 5, 2009
• Kodu: Visual Programming on the Xbox with P2P Level-sharing | by Brady Forrest on February 24, 2009