Everyblock's Code is Open-Sourced

The code for Adrian Holovaty's Everyblock has been released. The open-sourcing of the site's system were apart of the Knight News Challenge Program. Everyblock is a very impressive site that aggregates and geocodes local data -- news, crime, fire, restaraunt inspections and reviews - and then lets users define their interests down to the block-level.

Adrian made the announcement on 6/30. Here's the list of newly open-sourced, GPL'd goodies found on Everyblock's new Code page:

The main package (probably the thing you're looking for) is the publishing system, known as ebpub.
Second, the packages ebdata and ebgeo contain Python modules for processing data and making maps.
Third, the packages ebinternal and everyblock round out the code that powers EveryBlock.com. They're internal tools and are likely not of general use, but we're including them to be complete.
Finally, ebblog and ebwiki are our blog and wiki software, respectively. Because, dammit, the world needs another Django-powered blogging tool.

Django fans, Python geohackers and anyone who wants to build a local data aggregator are going to be thrilled. Adrian was one of the co-creators of Django and was one of the first Google Maps Mashup creators.

Everyblock has only launched in major US cities. There's plenty of room in the market for locals to create their own version. Everyblock spends a lot of time curating the incoming data feeds so I doubt that anyone will be able to roll out new sites too quickly. One thing to note: the trademark Everyblock is not available. However, the Everyblock team would not mind being acknowledged if you use their code. Personally I get a lot of value of Everyblock in my city. I get a daily email with all the crime, news and errata near my house.

Everyblock is now going to move onto the second stage of its existence. About five months Adrian blogged about the dilemna they would be facing when they open-sourced their software. As he said at the time:

But now we've reached an interesting point in our project's growth: our grant ends on June 30, and, under the terms of our grant, we're open-sourcing the EveryBlock publishing system so that anybody will be able to take the code to create similar sites. That's a Good Thing, in that EveryBlock's philosophies and tools will have the opportunity to spread around the world much faster than we could have done on our own, but it puts the six of us EveryBlockers in an odd spot. How do we sustain our project if our code is free to the world?

At the time I suggested that they try to federate with new everyblocks. After yesterday's announcement I mailed Adrian to ask him for a hint about their future plans, but for now he's keeping mum.

Four short links: 24 June 2009

Open Source Kids, Crowdsourcing Lessons, Flickr Secrets, Hadoop Spatial Joins

1. The Digital Open -- The Digital Open is an online technology community and competition for youth around the world, age 17 and under. Building a community of young open source hackers.
2. Four Crowdsoucing Lessons from the Guardian's Spectacular Expenses Scandal Experiment -- Your workers are unpaid, so make it fun. How to lure them? By making it feel like a game. "Any time that you’re trying to get people to give you stuff, to do stuff for you, the most important thing is that people know that what they’re doing is having an effect," Willison said. "It’s kind of a fundamental tenet of social software. … If you’re not giving people the ‘I rock’ vibe, you’re not getting people to stick around." (via migurski on delicious)
3. 10+ Deploys/Day: Dev & Ops Cooperation at Flickr -- John Allspaw and Paul Hammond's talk from Velocity. You tell any mainstream company in the world "10 deploys/day" and you'll be met with disbelief.
4. Reproducing Spatial Joins using Hadoop and EC2 -- bit by bit the techniques for emulating important operations from trad databases are being discovered and shared in the new database scene. (via straup on delicious)

Before and After Shots of Google's Iran Maps

There many places in the world where it is not possible for larger companies to map them. These can be for economic reasons as is the case for Black Rock City (the temporary 40,000 person home for Burning Man). Or for political reasons as is the case for Iran and countries such as China.
As I mentioned the other day Google greatly improved their map coverage of Iran via user contributions through their Mapmaker program. These user contributions were applied just a few weeks ago. Here are before and after screenshots of two Iranian cities. The before shot was taken on September 22, 2008; the after shots were taken on May 18, 2009.

Mashhad (Before and After)

Tabriz (Before and After)

Want a Map of Tehran? Use Open Street Map or Google

All eyes are on Tehran right now. As the center of the Iranian election protests the city has become increasingly important to websites this week. To keep their site up-to-date with this latest crisis area Flickr switched out the Yahoo road Map with Open Street Map. When I heard about this I wondered how other major mapping sites faired.

So I examined the road and satellite maps of Yahoo, Mapquest, Google, and Bing (formerly Live Maps). Looking at the images below it becomes very clear that user-generated maps win in hard to reach places. Both Open Street Map (above) and Google (below) rely on user-contributions. Open Street Map relies almost entirely on user uploaded GPS tracks for its mapping data across the world. After the jump i've included the satellite maps from each service (except for Mapquest who did not have them). They were

Google is using data acquired from their just-under-a-year-old Mapmaker program (Radar post). With Mapmaker users can add roads, POIs, regions and features. It's a very powerful tool that has greatly expanded Google coverage. Google has been slow and deliberate in using Mapmaker data on their main site. In fact it was just a couple of weeks ago that Iran's mapmaker data "graduated" to the main site. There are now 64 countries on Google that have been updated with Mapmaker data.

This isn't the first time Flickr has done this (Radar post). They've also used Open Street Map for Beijing, Black Rock City (2008), Tokyo, Buenos Aires, Adelaide, Sydney, Brisbane, Canberra, Melbourne, Baghdad, Kabul, Kinshasa, Mogadishu, Harare, Nairobi, Accra, Cairo, and Algiers.

So what's holding back Microsoft, Yahoo and Mapquest? Unknown, but hopefully they'll realize that their top-down approach isn't working.

Compare the Maps for yourself:
Note: I have included data layers where they were available (Google and Microsoft).

Google Maps:

(The markers include Wikipedia articles, photos, video, webcams, POIs, and public transit stations)

(The markers include Photosynths, user collections, photos and Wikipedia articles)

Yahoo Maps:

Mapquest Maps:

(This took a while to find, I had to find the International Maps page and click-thru a couple more pages to get the map)

(continue reading)

XKCD on the Future Self

This morning's XKCD, Latitude, spells out one of the reasons people will be weary of setting up continuous location trackers: the future self. The future self forgets that they are sharing their location and then act as if no one knows where they are going. In this case Megan's friend tracks her stops at a sex shop, toy store, hardware store and finally the burn ward, telling a pretty clear story of a mistranslated kama sutra (ahem).

The name Latitude comes from the recent Google service that will share your location with your network (or publicly). However, it could have just as easily been called Loopt, Brightkite, Fire Eagle or any number of other location-updating services. I am personally looking forward to these services becoming ubiquitous so that I can track my location and aspects of my life, but as they currently stand these services are not poised for mainstream adoption. They need to do more to prevent people from embarrassing or endangering themselves.

One location-sharing service that takes a more constrained role is Glympse. It allows you to share your location on an ad hoc basis with specific people for a specific amount of time. Check-in services like FourSquare (or Dodgeball) let you specify when you self-locate as you wish. These are half-measures and don't meet everyone's needs, but they provide important steps in the right direction.

What the iPhone 3GS and 3.0 OS Means for Geo Devs

Yesterday's announcements around the iPhone 3GS and new 3.0 OS were significant to consumers and developers. Here are some of the changes that will make geo devs happy.

Google Maps Views (Mapkit) - Developers can now take advantage of Google Maps within their apps. This means that you no longer have to building your own mapping system for your native apps. Companies like Platial and Pelago who made huge resource investments in building their own maps early on that did will now have to decide if they should continue the maintenance or go with Google.

Dynamic Map Markers - Map markers in mapping apps can now be changed based on realtime data on the server. This is used in the featured ZipCar app (screenshot) to show which cars are available to use at a certain time.

Safari - The new Safari will be faster, allow for offline content and provide access to location APIs. This is huge. It means that a web application can find a user's location. Google showed a version of Latitude using this at Google I/O. The faster browser also means that developers can have richer apps with less performance fears. Finally, the ability to have offline access will allow content to be cahced. Unfortunately, there is no word if Safari will have access to the Compass (I doubt it) and it still does not have access to the Camera (a key component to many geo apps).

In-Application Purchases - iPhone 3.0 should also bring the ability to purchase content/items from within an application. Developers will be able to sell geo data and upgrade app services with this feature.

Here are some iPhone 3GS goodies that are available due to the new hardware and are only available on the iPhone 3GS.

Camera - The new camera is 3MP and has video capabilities. Awesome, but more importantly for geo developers it has an auto-focus macro-mode. This will allow users to take pictures of small Barcodes and QR Codes. The previous iPhone camera had issues reading them. These physical marks (QR Codes especially) are often used to convey digital information like an URL or email address.

Compass - The digital compass provides orientation to the phone. With orientation apps that layer virtual information (AKA Augmented Reality) over the camera view become possible. This is a great boon to the iPhone. Hopefully we'll see some great designs come forward. No word on whether or not Safari will have access to the compass.

Peer-to-Peer (GameKit) - This API lets two apps talk to each other over BlueTooth. It is designed for gamers, but if the API is open then it will be used by geohackers to do proximity checks (who is nearby me) and for geocontent sharing.

Accessories API - Apple has released the ability for third-party hardware to interact with the iPhone. This is going to be a huge boon to hardware hackers everywhere. The iPhone can take in and transmit sensor and location data for them. Nike won't have to go through this API; support for Nike+ is built-in giving all hardware hackers something to dream about.

Graphics - The new phone will have improved performance and increased 3D support. You mostly hear news about how this will affect games, but awesome 3D geo apps like Google Earth and UpNext will also get the 3D benefits.

Even with all these goodies, there are some obvious carrots that are out of reach for developers. Some of them are:

Find My iPhone - If you have MobileMe iPhone 3.0 users can find their phones, send a message to their phone and perform a data wipe on their iPhone. This is a service; not an API. It will probably get Apple a lot of MobileMe accounts. Apple did not provide any way for apps to passively get a user's location, however they obviously have the ability to get that info.

Tomtom Turn-By-Turn Navigation - Apple forbid third-party developers from developing turn-by-turn navigation, obviously waiting for a big partner to take on the task. Tomtom won the honor.

There were of course some disappointments:

Push Notifications - Apps cannot send data to the server unless they are open. That means that location data cannot be sent to the cloud in the background. This is a big disappointment for those of us who want Latitude, Fire Eagle, Loopt, Whrrl, BrightKite to be able to track our every move. Maybe it will come in a future release.

I've discussed my disappointment about the lack of background location support before. Read Apple's Big Location Chance, Or When Is The iPhone Going To Use That GPS? for an expanded read on why Apple should do this.

Thanks to Raven for your help on this post and Ryan Block for the gdgt photos from WWDC.

Mapumental: Time & Scenicness in Maps

MySociety has given us a sneak peak at Mapumental, a map app that lets you pivot on travel-time, "scenicness", and house-price in the London area. Just enter a postal code and if you're looking for a home in the area Mapumental should be very helpful to you. It is an update to a previous foray into temporal maps (you can try it out on the embed in a Radar post of mine).

The map has a slider to control each of the dimensions. The time slider lets you choose how long you want to commute to get to the desired postal code by 9AM. This is a big change from the original map, which let you . The scenic slider lets you determine how nice of a place you want to live in. The housing price map The map above is quite depressing. It shows the areas you could live that will take you less than an hour and a half to get to work, has housing available for 750K GBP and is just barely scenic at level 3. Your options open up a lot if you ignore scenicness for this zipcode.

To make a useful and useable map with this many controls and data points is difficult. The base maps come from Open Street Map. The travel data (rail, bus, ferry) comes from Traveline (National Public Transport Data Repository); it's all based on a Tuesday in October, 2008. The map tiles, sliders and overall UI was inspired built by Stamen Design. MySociety's travel-time maps were pioneered by Chris Lightfoot.

The most unique dataset included in Mapumental is "scenicness". The data was gathered by user votes in the web app Scenic Or Not. As Tom told me, "We have 173,816 1*1 km squares voted on at least 3 times each, by different people. We're 80% of the way to a full dataset. Data comes from Geograph."

They've got more planned for it like making it public, allowing for alternate arrival and departure times, and allowing for multiple destinations to allow for couples. Those are all great additions, but what I would really like to see is either open-sourced code or an API so say that other geographic areas could have this functionality. How would this change the real estate market? If you could see this style of map for any industrialized area would it change the way you think about your quality of life and what it costs. Sounds like a great task for real estate sites Zillow or Trulia, alternately it's something that Walkscore could tackle.

Right now Mapumental is in Beta, but if you want early access Tom Steinberg sent a hint to BoingBoing: "Beta's private at the moment but we're handing out invites in exchange for declarations of love." Send your missives here. To learn even more about the project check out the short video after the jump.

(via Cory @ Boingboing)

Updated: Properly attributed Stamen's role on the project

(continue reading)

Four short links: 1 June 2009

Spymaster, Arsenic, Maps, and Happiness

1. Spymaster -- a faux-spy game on Twitter: Each player becomes a master of a spy ring based upon their Twitter followers list. The more people that follow you and are playing characters in Spymaster, the more powerful your network will be. As a spymaster, you can perform tasks or attack other spymasters on Twitter. With each successful attempt, you will gain virtual currency and points that allow you to grow even stronger. I'm nervous that it's a project of a classified ads company, but intelligent friends appear to be enjoying it, but that may just be be the jaded eye of a world-weary veteran of pyramid schemes and spamalots.
2. Getting Arsenic Out Of Water -- MIT Technology Review piece about the IBM discovery that a chemical used to pattern chips also acts as a membrane to remove arsenic. More stuff that matters. (via roterhund on Twitter)
3. Mapumental -- MySociety folks making maps useful. It's the continuation of time travel maps, where bus, train, tram, tube, and ferry timetables are mashed with real estate prices to show you where you can live for what you can afford and how long a commute you want. A new twist is crowdsourced "how scenic is this area?" data, so you can choose other dimensions for where you might want to live. New dimensions on transportation data and travel planning.
4. What Makes Us Happy? (The Atlantic) -- the real world is a lot more complex than trivial "get happy fast!" self-help books would have you believe. This longitudinal study shows how complex happiness and misery are. Vaillant’s other main interest is the power of relationships. “It is social aptitude,” he writes, “not intellectual brilliance or parental social class, that leads to successful aging.” Warm connections are necessary—and if not found in a mother or father, they can come from siblings, uncles, friends, mentors. The men’s relationships at age 47, he found, predicted late-life adjustment better than any other variable, except defenses. Good sibling relationships seem especially powerful: 93 percent of the men who were thriving at age 65 had been close to a brother or sister when younger. In an interview in the March 2008 newsletter to the Grant Study subjects, Vaillant was asked, “What have you learned from the Grant Study men?” Vaillant’s response: “That the only thing that really matters in life are your relationships to other people.” (via timoreilly on Twitter)

New Geo For Devs From Google I/O

Today at Google I/O, Google has made several announcements for geo developers. To sum:

• Google is updating (not abandoning!) its Flash API, but it still prefers the Javascript one
• Google is pushing the Maps API into mobile (and performance is a big part of the push)
• Geolocation is going to be a part of every Google product eventually
• Android is being backed by deep pockets
• Google is preparing an army of Qualified Developers to bring more them more API customers.

More details are below. I chatted with Pamela Fox of the Google Geo team about the announcements.

Flash API with 3D - This upcoming release adds the ability to add a 3D perspective to Google's existing Flash Maps API. It's not a full blown implementation of the Google Earth API PLugin which provides Google Earth level 3D in the browser. Instead this release allows you to tilt the maps and play with perspective. The screenshot above shows how you can tilt the map on its canvas. As Pamela said during the demo "we hope it's useful when you need 3D, but the Google Earth's 3D is overkill." (paraphrased - she said it much better in the session)

Qualified Developer - Google has expanded their Qualified Developer program to include Maps. Very soon the team will be releasing a semi self-serve process for getting the Google seal of approval on your Google Geo API skills. The qualification process will be a test (in the process of being written), referrals, sample code and your participation in the developer forums. Your qualification status will help you stand out in the Google Solutions Marketplace.

Map API v3 - The Maps team also chose Google I/O as the place to launch the new version of their Javascript Maps API. The key improvement is in the performance area and a reduction in initial download size. The other big push was in the mobile arena. Both Chrome and iPhone Safari mobile are supported (Android mobile still has some bugs though). They've also done an overhaul of the Geocoding API (good!) and enabled the default update of the mapping controls on third-party sites.

Client Location on Google Maps - At both Google I/O and Where 2.0 Google showed the ability to locate yourself when on Google Maps. By default it geolocates you by IP, but if you have Gears it will take advantage of Gear's Geolocation API. The feature will show up in the mapping controls as a blue dot. No word on when this feature will be released. I hope this feature eventually becomes available in the Maps API.

Android Developer Challenge V2 - Google has announced V2 of its Android Developer Contest. They'll be awarding prizes across multiple categories including geo-friendly ones like Travel and Productivity. The winners will be chosen mid-November.

Of these announcements I find the Qualified Developer program to be the most significant. By certifying developers Google will be enabling developer to get more Google more API customers. The program started last year with Gadgets Ads. Once they work the kinks out I am sure that this program will extend to every API that Google has and will be quite the moneymaker for the participating devs.

Four short links: 27 May 2009

Hacker Browser, Design and Engineering, Twitter Data, Fire Eagle Updater for OS X

1. uzbl -- lightweight WebKit-based web browser controlled with vim-like keystrokes, controllable through a FIFO for scripting, and all the "features" (bookmarking, history, changing URL) happen through external scripts. For the hardcore. (via joshua on delicious)
2. A Conversation With Eric Rodenbeck About Usefully Cool Design and Engineering (Jon Udell) -- if we could only distil Stamen down to their barest essence, we could make a fortune selling it on the black market ...
3. Twitter Data -- using Twitter as a conduit for messages that have semantic markup. My gut reaction is that I'd prefer pure JSON in the data tweets, because a hybrid gives you poor use of the limited bandwidth and there seems no strong reason to care about human readability. (via Ted Leung)
4. Clarke -- elegant OS X updater for Fire Eagle that uses Skyhook to determine your location.

Yahoo! Placemaker - Open Location, Open Data and Supporting Web Services

Today at Where 2.0 Tyler Bell, the Head of Yahoo's Geo Technologies Group, launched Placemaker (this link should be live at posting). Placemaker is a webservice that takes in text and returns the locations found within via either XML or enhanced GeoRSS. The locations Placemaker returns come in the form of WOEIDs (Radar post). You might be cautious about relying on Yahoo's ID system for your locations. To alleviate your fears Yahoo! is announcing the release of GeoPlanet Data, all of the WOEIDs available as a free download under Creative Commons in June. Woot!

Placemaker's geoparsing API will return WOEIDs and place names for all of the locations detected in the submitted text. This text can be structured or unstructured. If their are multiple locations detected the it will return a common ancestor called the Doc Scope. For example if San Francisco and Los Angeles are in the text then the Doc Scope will be "California". If San Francisco and Sacramento were in unstructured text then the Doc Scope would return the colloquial term "Northern California". There are no explicit limits on the API as long as your usage is "nice" -- if it's not you may find yourself shut off for a while.

Placemaker is an updated version of the geoparsing engine currently available through Yahoo! Pipes. This release rightfully makes geoparsing a stand-alone API. If you want to learn more about Placemaker Yahoo has posted the following instructions:

1. Read the online documentation at developer.yahoo.com/geo/placemaker/guide
2. Get an Application Id at developer.yahoo.com/wsregapp
3. POST your content to wherein.yahooapis.com/v1/document

The WOEIDs will be made available under the CC-Attribute license. It will ultimately include over 5 million entities in multiple languages. Relationships between the entities will be included.

Up till now Geonames IDs have been used as place IDS by many apps. All of Geonames' data is freely available for download. It was tough for Yahoo to compete with this open data solution. Today's release and announcement really ups the game. By making the data freely available developers will no longer have much fear about using the data. WOEIDs were first released as a webservice a year ago. At that point in time I expect the free release of the WOEID data to greatly increased the uptake of these supporting webservices and make Yahoo an integral part of mapping mashups.

Yahoo! has more info over on their Geo Technologies Blog.

Four short links: 18 May 2009

Scientists, Scammers, Satellites, and Safe Havens

1. Scientists Without Borders -- "Mobilizing Science, Improving Lives". mobilize and coordinate science-based activities that improve quality of life in the developing world. The research community, aid agencies, NGOs, public-private partnerships, and a wide variety of other institutions are already promoting areas such as global health, agricultural progress, and environmental well-being, but current communication gaps restrict their power. Organizations and individuals do not always know about one another's endeavors, needs, or availability, which limits the ability to forge meaningful connections and harness resources. This situation is especially striking in light of the growing realization that integrated rather than focused approaches are crucial for addressing key challenges such as extreme poverty and the glaring health problems that accompany it. See also Geeks Without Borders, but is there anyone running a program that sends geeks into the field where they're needed? I know a lot of open source folks who have been volunteering around the world in poor nations, but I haven't found a site that coordinates this. Can anyone point me to such a thing?
2. The Psychology of Being Scammed -- UK government report into the psychology of scammers' victims. Lots of insights into successful scams (parallels drawn to finance or startups left as exercise to reader) and some counter-intuitive findings like Scam victims often have better than average background knowledge in the area of the scam content. For example, it seems that people with experience of playing legitimate prize draws and lotteries are more likely to fall for a scam in this area than people with less knowledge and experience in this field. This also applies to those with some knowledge of investments. Such knowledge can increase rather than decrease the risk of becoming a victim. (via Mind Hacks)
3. GPS Accuracy Could Start Dropping In 2010 (Tidbits) -- the Air Force has had difficulty launching new satellites. The GAO has calculated - using reliability curves for each operational satellite - that the probability of keeping a 24-satellite constellation in orbit drops below 95 percent in 2010, and could drop as low as 80 percent in 2011 and 2012. (via geowanking)
4. Open Database Alliance -- an attempt to provide a safe home for MySQL given the Oracle acquisition of Sun. [...] a vendor-neutral consortium designed to become the industry hub for the MySQL open source database, including MySQL and derivative code, binaries, training, support, and other enhancements for the MySQL community and partner ecosystem. The Open Database Alliance will comprise a collection of companies working together to provide the software, support and services for MariaDB, an enterprise-grade, community-developed branch of MySQL.

Recent Posts

• Come to Ignite Where & Launchpad | by Brady Forrest on May 13, 2009
• That Was Fast: Mapme.at Uses Latitude API | by Brady Forrest on May 6, 2009
• Google's Sneaky Launch of Latitude's Location-Sharing API | by Brady Forrest on May 6, 2009
• Swine Flu Tracker | by Brady Forrest on May 5, 2009
• Jack Dangermond Interview 3 of 3: The Geoweb | by Brady Forrest on May 1, 2009
• Where Week 2009 | by Brady Forrest on May 1, 2009
• Jack Dangermond Interview 2 of 3: Sharing Government GIS Data | by Brady Forrest on April 30, 2009
• Jack Dangermond Interview 1 of 3: Web Mapping | by Brady Forrest on April 29, 2009
• Locavore's Open Data | by Brady Forrest on April 23, 2009
• Where 2.0 Preview - DARPA's TIGR Project Helps Platoons Stay Alive | by James Turner on April 21, 2009
• A Telling Map of Job Losses | by Brady Forrest on April 16, 2009
• Where 2.0 Preview - Building the SENSEable City | by James Turner on April 16, 2009