Robots.Txt and the .Gov TLD

by Carl Malamud | @CarlMalamud | comments: 4

I'm on the board of CommonCrawl.Org, a nonprofit corporation that is attempting to provide a web crawl for use by all. An interesting report just got sent to us about the use of robots.txt files within the .Gov Top Level Domain, a standard known as the Robots Exclusion Standard.

In examining about 32,000 subdomains in .gov, it turns at least 1,188 of these have a robots.txt file with a "global disallow," meaning robots are excluded from indexing this content. Even more curious, on 175 of these sites, while there is a global disallow, there is a specific bypass that allows the Googlebot to index the data. You can look at the raw data on Factual.

At Public.Resource.Org, we've always felt that the use of a robots.txt file by the government should only be used for purposes of security and integrity of the site, not because some webmaster arbitrarily decides they don't want to be indexed. Indeed, on several occasions we have deliberately ignored government imposed robots.txt files because we felt this was an arbitrary and illegal attempt to keep the public out.

And, needless to say, it doesn't make any sense at all to let in some webcrawlers and not let in others. If this is a reaction to a security/integrity issue, such as limited capacity, the proper thing to do is include in the robots.txt file a comment that can be used by other bots to explain what is going on. For example, it could be perfectly reasonable for a government group faced with limited capacity to ask a robot to limit crawls to a certain number of queries per second and only whitelist crawlers that agree to that condition.

Government webmasters should use the robots.txt file sparingly, and should do so in a non-discriminatory fashion.

Asia Continues to be Facebook's Strongest Growth Region

by Ben Lorica | @dliman | comments: 0

With Facebook topping 330 million active users over the past week, the company's strongest growth region continues to be Asia. Over the last 12 weeks, Facebook added close to 17M active users in Asia alone. Since my previous post, the share of active users from Asia grew by 2% (to 13.5% of all users), and roughly 1 in 7 users now come from the region. With a market penetration under 2%, Facebook is poised to add many more users in Asia (and Africa).

Compared to the U.S., the proportion of Facebook users in their teens (13-17) or in the 18-25 age group are much higher in Asia:

As was the case in other parts of the world, expect the share of users 45 and older to climb as Facebook becomes more mainstream in Asia. Growth was strong across all age groups in Asia over the last 12 weeks, particularly among teens (+90%) and the 18-25 age group (+60%).

In other regions, notably North America, Europe, the Middle East, and South America, growth in the 18-25 age bracket, lagged behind users 45 and older.

In closing I want to highlight countries (within several regions) where Facebook has been growing rapidly:

(continue reading)

Four short links: 20 November 2009

Social Network Search for Morons, Bulking Up Bio Data, Better E-Mail, Better Standards

by Nat Torkington | @gnat | comments: 1

1. Spokeo -- abysmal indictment of society, first prize in mankind's race to the bottom. Uncover personal photos, videos, and secrets ... GUARANTEED! Spokeo deep searches within 48 major social networks to find truly mouth-watering news about friends and coworkers. PS, anybody who gives their gmail username and password to a site that specializes in dishing dirt can only be described as a fucking idiot. (via Jim Stogdill, who was equally disappointed in our species)
2. Biologists rally to sequence 'neglected' microbes (Nature) -- The Genomic Encyclopedia of Bacteria and Archaea is project to sequence genomes from more branches of the evolutionary tree of life. Eisen's team selected and sequenced more than 100 'neglected' species that lacked close relatives among the 1,000 genomes already in GenBank. The researchers reported earlier this year at the JGI's Fourth Annual User Meeting that even mapping the first 56 of these microbes' genomes increased the rate of discovery of new gene and protein families with new biological properties. It also improved the researchers' ability to predict the role of genes with unknown functions in already sequenced organisms. (via Jonathan Eisen)
3. Mail Learning: The What and the How (Simon Cozens) -- a few things that a really good mail analysis tool needs to do. I hope that my mail client and server does these out of the box in the next five years.
4. Introducing the Open Web Foundation Agreement -- The Open Web Foundation Agreement itself establishes the copyright and patent rights for a specification, ensuring that downstream consumers may freely implement and reuse the licensed specification without seeking further permission. In addition to the agreement itself, we also created an easy-to-read "Deed" that provides a high level overview of the agreement. Applying the open source approach to better standards.

Health gets personal in the cloud

Google Health Beta and Microsoft's My Health Info

by Brian Ahier | @ahier | comments: 13

Healthcare is one of the biggest industries in the world. The United States spends over 17% of its GDP on healthcare and the issue of the industry's future is being hotly debated in Congress. Whatever happens to other elements of health reform, health information technology will play a key role in moving us towards the goal of bending the cost curve and improving quality and clinical outcomes. A Personal Health Record (PHR) is one way that patients can have some control of their own health data, while providing an interoperable platform for sharing relevant clinical data between providers. Healthcare is changing rapidly and there are some important trends worth watching.

Healthcare in the near future will be quite different than it is today. Web enabled technology is already changing the way medicine is practiced. As the digital nation comes of age we will see new opportunities, and new challenges, bringing healthcare in America into the 21st century. Health consumers will come to expect they will have control over their own health data. Having secure, interoperable access to clinical data will allow patients to partner with their care providers in new ways incorporating Web 2.0 principles.

For example, Google announced at the Health 2.0 conference that they have entered into a partnership to provide telehealth services through their Google Health platform using MDLiveCare. With the integration of MDLiveCare technology, Google can provide a service that offers patients access to doctors from remote locations, via webcam or telephone, into its personal health record offering. This will be particularly valuable for those who are caring for their loved ones from far away. My family is scattered around the country and caring for our mother with advanced stage Alzheimer's was quite a challenge that would have benefited from this type of service. Here is a screenshot of Google Health:

"Patients remember less than 25% of what they're told when they consult with a doctor,” said Bob Smoley, CEO, MDLiveCare, in the statement. "By directly synchronizing the information that's shared…we're able to provide patients with a convenient solution to review their physician or therapist encounters."

(continue reading)

Four short links: 19 November 2009

Chumby One, Gorgeous IE Debugger, Freer Than Free, and Phone-a-Friend for Government IT

by Nat Torkington | @gnat | comments: 0

1. Chumby One (Bunnie Huang) -- new Chumby product released. In addition to being about half the price of the original chumby, the new device added some features: it has an FM radio, and it has support for a rechargeable lithium ion battery (although it’s not included with the device, you have to buy one and install it yourself). There’s also a knob so you can easily/quickly adjust the volume. But I don’t think those are really the significant new features. What really gets me excited about this one is that it’s much more hackable.
2. Deep Tracing of Internet Explorer (John Resig) -- very sexy deep inspection of Internet Explorer. Finally, something IE does better than Firefox (other than exploits). dynaTrace Ajax works by sticking low-level instrumentation into Internet Explorer when it launches, capturing any activity that occurs - and I mean virtually any activity that you can imagine. (via Simon Willison)
3. Less Than Free -- begins by talking about Google giving away turn-by-turn directions on Android, and then analyses Google's "less than free" business model: Additionally, because Google has created an open source version of Android, carriers believe they have an “out” if they part ways with Google in the future. I then asked my friend, “so why would they ever use the Google (non open source) license version.” Here was the big punch line - because Google will give you ad splits on search if you use that version! That’s right; Google will pay you to use their mobile OS. I like to call this the “less than free” business model. This is a remarkable card to play. Because of its dominance in search, Google has ad rates that blow away the competition. To compete at an equally “less than free” price point, Symbian or windows mobile would need to subsidize. Double ouch!!
4. Expert Labs -- a new independent initiative to help policy makers in our government take advantage of the expertise of their fellow citizens. How does it work? Simple: 1. We ask policy makers what questions they need answered to make better decisions. 2. We help the technology community create the tools that will get those answers. 3. We prompt the scientific & research communities to provide the answers that will make our country run better. New non-profit from Anil Dash.

Four short links: 18 November 2009

Web Time Travel, UK Map Data Liberation, Streetview Mashups, 3D Retail

by Nat Torkington | @gnat | comments: 0

1. Memento: Time Travel for the Web -- clever versioning hack that uses HTTP's content negotiation to negotiate about the date!
2. Ordnance Survey Maps to Go Online -- The prime minister said that by April he hoped a consultation would be completed on the free provision of Ordnance Survey maps down to a scale of 1:10,000, (not the scale of a typical Landranger map set at 1:25,000). The online maps would be free to all, including commercial users who, previously, had to acquire expensive and restrictive licences at £5,000 per usage, a fee many entrepreneurs felt was too high. No word yet on license. (more details here)
3. Mapsicle -- open source Javascript library to create mashups and application on Google Streetview, from NZ developers Project X. It has been released by Google as part of the Maps Utility library.
4. Freedom of Creation Shop -- online store for 3D-printed objects. (via Makezine).

The iPhone: Tricorder Version 1.0?

by James Turner | comments: 4

The iPhone, in addition to revolutionizing how people thought about mobile phone user interfaces, also was one of the first devices to offer a suite of sensors measuring everything from the visual environment to position to acceleration, all in a package that could fit in your shirt pocket.

On December 3rd, O'Reilly will be offering a one-day online edition of the Where 2.0 conference, focusing on the iPhone sensors, and what you can do with them. Alasdair Allan (the University of Exeter and Babilim Light Industries) and Jeffrey Powers (Occipital) will be among the speakers, and I recently spoke with each of them about how the iPhone has evolved as a sensing platform and the new and interesting things being done with the device.

Occipital is probably best known for Red Laser, the iPhone scanning application that lets you point the camera at a UPC code and get shopping information about the product. With recent iPhone OS releases, applications can now overlay data on top of a real time camera display, which has led to the new augmented reality applications. But according to Powers, the ability to process the camera data is still not fully supported, which has left Red Laser in a bit of a limbo state. "What happened with the most recent update is that the APIs for changing the way the camera screen looks were opened up pretty much completely. So you can customize it to make it look any way you want. You can also programmatically engage photo capture, which is something you couldn't do before either. You could only send the UI up and the user would have to use the normal built-in iPhone UI to capture. So you can do this programmatic data capturing, and you can process those images that come in. But as it turns out, at the same time, shortly after 3.1, the method that a lot of people were using to get the raw data while it was streaming in became a blacklisted function for the review team. So we've actually had a lot of trouble as of late getting technology updates through the App Store because the function we're using is now on a blacklist. Whereas it wasn't on a blacklist for the last year."

Powers is hopeful that the next release of the OS will bring official support for the API calls that Red Laser uses, based on the fact that the App Store screeners aren't taking down existing apps that use the banned APIs. Issues with the iPhone camera sensors pose more of a problem for him. "In terms of science, it's definitely a really bad sensor, especially if you look at the older iPhone sensor, because it has what's called a rolling shutter. A rolling shutter means that as you press capture or rather as the camera is capturing video frames or as you capture a frame, the camera then begins to take an image. And it takes a finite number of milliseconds, maybe 50 or so, before it is actually exposed to the entire frame and stored that off into a sensor. Because it's doing something that's more like a serial data transfer instead of this all at once parallel capture of the entire frame, what that causes is weird tearing and odd effects like that. For photography, as long as it's not too dramatic, it's not a huge deal. For vision processing, it's a huge deal because it breaks a lot of assumptions that we typically make about the camera. That has gotten better in the 3GS camera, but it's still not perfect. It is getting better, especially when the camera's turned on the video mode."

(continue reading)

What Does Innovative Social Engagement Look Like For Businesses and Governments?

by Mark Drapeau | @cheeky_geeky | comments: 26

I've been thinking about the topic of Government 2.0 a lot lately. Part of this topic deals with the multi-directional engagement between government and citizens. This is what the White House and others have termed a more transparent, collaborative, and participatory government.

Unfortunately, the engagement for the most part is not very authentic nor meaningful. Boring "fan pages" on Facebook are one example I've written about, but there are many others. Often, engagement, when it does happen has so many rules associated with it, or such a high barrier to entry, or such a limited window as to be practically meaningless.

It seems to me that everyone can celebrate the fact that government entities merely have a YouTube channel here, a Twitter account there, or a Blogger profile some other place (the so-called "TGIF revolution"), or we can think a little harder about what the goals of citizen engagement really might be, and how to go about achieving them. But first, a personal example of responsiveness and engagement from the private sector.

On the evening of Nov 2nd, I tweeted from my phone about a local DC restaurant, Co Co Sala, just as I was leaving. We had a nice experience, but the hostess had been a little, shall we say, disinterested in helping us? So I commented as much.

Less than a week later, the co-owner of Co Co Sala sent me an email and cc'd his general manager. He apologized for the treatment I experienced, assured me it was not policy, introduced me to the manager, and said he'd talk to his staff. It was a four-paragraph email. I've never met him before, and furthermore, my personal email is discoverable but not the most easy thing to find.

This is what real social innovation looks like. This is what customer service looks like. This is what true engagement with stakeholders looks like. I want to give this great lounge Co Co Sala a hearty shout-out for not only having a great product, but also really caring about their customers.

Now, imagine we weren't talking about a restaurant here. Imagine we are talking about the Department of Motor Vehicles, or the Patent and Trademark Office, or your Congressman. If you tweeted, would they see it? Would they care? Would they react in any way? I think the answer in many cases is no. And when was the last time you gave the DMV a shout-out for a job well done?

Let's look at a sliver of data. According to TweetStats.com, the people behind the White House Twitter account reply to individuals less than 2% of the time, and seem to have never @ replied to any single more than once (i.e., they have never come close to a conversation). They re-tweet others' tweets about 6.5% of the time, but they only seem to re-tweet other government accounts and the New York Times. Granted, there are more people tweeting about White House issues than Co Co Sala, but does the above data represent any caring in any way, shape or form?

The terrific techPresident blog recently noted that actor Vin Diesel is the single most followed living person on Facebook - and that he recently passed up President Obama. Perhaps that's because Vin Diesel's Facebook fan page is awesome. He is engaged, his fans are engaged, and the tone is informal and fun. There are also many other high-profile people who have taken the plunge into innovative social engagement; my favorite at the moment is Alyssa Milano.

So when exactly did "serious and formal" become a substitute for "informative and meaningful" in government circles? And why is everyone scared of letting their guard down in public? People and entities that innovate and use new social networking tools to engage with stakeholders will be winners. The ones that don't will be losers in the long run. It's that simple.

If a goal of Government 2.0 is to provide citizens better services, and a strategy towards reaching that goal is to use social media tools to communicate better with citizens on multiple channels, it seems to me that listening and responding better to comments and complaints would be a great tactic.

The reason why people still cite the TSA's blog as a good example of citizen engagement is because few other outstanding examples of federal government social media engagement seem to have emerged in 2009. What does 2010 have in store?

It is somewhat outside the scope of this post, but my guess is that more and more local government responsiveness and engagement is happening. We heard some of those stories at the Gov 2.0 Expo Showcase in September. What are some new ones that the feds should hear about?

Four short links: 17 November 2009

Digital Natives, Supersexy C64 Debugger, a Google Tripwire, and a Patient Botnet

by Nat Torkington | @gnat | comments: 1

1. Digital Natives (Ze Frank) -- digital natives have grown up in a landscape where access to information and influence has been flattened. they have watched media distribution bottlenecks in the form of networks and studios lose influence to youtube and independent production houses. They have watched companies bow down to viral video critiques, and watched political systems get hacked by social networks. this is a generation that doesn't understand restrictions on access to media if those restrictions are inefficient or obviously detrimental to the system as a whole. this is a generation that has been at war with DRM and copyright right from the start. it is a generation awash with free tutorials and download-able source code. When is a conversation with a precocious 17 year old a glimpse into an inter-generational gulf with implications for the role and status of formal education, and when is it just an encounter with a brat? Ze's piece is worth reading, whichever way it comes out.
2. ICU64 -- an open source Commodore 64 emulator (Frodo) hacked to visually and textually display memory. Watch the video embedded below, it's hypnotic and seductive. It immediately made me want one for my programs (without having to port my code back to 6502 assembler). (via waxy whose return from pneumonia is greatly welcomed)
3. Me and Belle du Jour -- interesting story from a UK blog master who guessed her identity but kept it secret, creating a googlewhacked page as a tripwire to let him know when someone else guessed. He tipped her off that her cover was blown. (via waxy again)
4. The Hail Mary Cloud -- the world's slowest yet effective brute force attack. If you publish your user name and password, somebody who is not you will use it, sooner or later. A botnet is brute-force trying every known username and password combination against every known ssh server. Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.

Turning Predictions into Opportunities

by Nat Torkington | @gnat | comments: 7

The view from the eye of a recession isn't great. When companies are going bust, unemployment growing, and everyone's scouring their budgets for costs to cut, it can be hard to see opportunities. However, when Tim pointed to Stephen O'Grady's fine set of 2010 predictions I found myself popping with "oh, so naturally this will happen next ..." thoughts. Think of this as a glimpse of the blue sky after the economic funnelspout that's demolished our economy. (Continuations of the tornado metaphor with "being sucked into the cloud", or "trailer park economics", or "we're not in Kansas any more, Tantek" left as an exercise to the reader)

1. As every cloud provider creates their own "open API" (itself a fraught term), look to see the rise of brokers who can migrate you from one cloud to another. Deltacloud is an early free project here from RedHat, but there are many business opportunities waiting. It's possible that companies will pay for assurance (you've tested your migration tool, you know it works on corner cases), service vs product (they don't want tools to run, they want to pay you to install and maintain the tools accessible through a web console), or premium services so that you're a partner helping them get the most from the cloud and not simply a vendor.
2. We're a long way from sated in the world of collaboration tools. The current rage is mail learning, applying machine learning techniques to email so as to better understand social networks and prioritise incoming email messages and these are largely server-based solutions because it's so hard to get access to the desktop/web clients. Should Google Mail create an app store environment with hooks into the backend, the game could be on for consumer plays around email analytics, prediction, and simply smarter behaviour (why does my email client still not tell me when I say "see attached" yet don't have an attachment in the message?).
3. Beyond email, many interesting tools have sprung up around the Gov 2.0 space that have applicability within organisations. Yammer has done well to bring Twitter to large companies, but there are still opportunities around simple document markup and suggestion gathering and filtering. Solve a real problem and there's money waiting.
4. Google's low overhead management is made possible by its automated intranet and the visibility into projects from public code repositories, public smoke builds, and public status blogs. The opportunities to sell this into large companies looking to be "more like Google" are huge.
5. If Stephen's right that datasets are increasingly viewed as "serious, balance sheet-worthy assets" then the world is going to need some serious balance sheet-worthy help in valuing those assets.
6. Big data is being democratised, but there's a lot of unmet need in businesses around data warehousing. The typical solution is to build a data warehouse team around a product like Oracle, but I've heard plenty of business people grizzling about the result. They want answers, they don't want the headaches and lag that a data warehouse involve. Big Data (or Cloud Analytics or whatever) may be the opportunity to figure out a new minimum viable product for these folks, and offer it without the "data warehouse" baggage. This might be back end, might be UIs, might be visualisation, but all of these have a lot of room for improvement.
7. The proliferation of developer targets immediately makes me think of the early PC era. It makes sense to proliferate: let the most useful ("successful") bubble to the top and survive naturally. At this point in the evolution of the scaleout of massively multiplayer online programming languages, we don't know exactly what winning looks like: it's a big feedback loop between the people who build the programming languages and the people with problems to solve (there are always more of the latter than former) and each time we go around it we know more about what is and isn't useful in this brave new world of coding for other people's data centres. Opportunity? Join the mob and write your own programming language, or simply take your commercial opportunity for a spin around the many different languages out there and be the first in your niche to find a good fit between problem space and solution tool.
8. Stephen's throwaway comment "I’ve never subscribed to the idea that only what can be measured can be managed - open source, in particular, belies that claim" seems like a thrown gauntlet on open source analytics. In particular, I suspect there's a tools opportunity around the nebulous "community manager" role that every company seems to need. It's part CRM, it's part developer tool, it's part tech support, and part camp mother. Usefully quantify aspects of open source development and help companies that are doing it to know how they're doing and what they could do better.
9. Marketplaces are big in mobile, but I look to other areas as ripe for the picking. For example, if Google Apps are catching on in many companies then a plugin marketplace is a natural extension. It would build out the Apps suite faster than Google can, would enable the tight loop between demand and supply that will drive the product along, and make Google's offering very different from other parties. This is also true of Microsoft and others, but I feel like momentum is more with Google's product than the others. (A feature can push a leader further in front, but rarely helps a laggard leapfrog to the lead)
10. Every marketplace thus far has been flawed. Apple's famously annoys many developers and blocks huge categories of product (the "don't be better than we are" rule, which is hard to justify as being in the customer's interest), but don't forget Palm's impedance mismatch with jwz's open source code. I think the final chapter on how marketplaces work is far from written.
11. NoSQL tools remain in their infancy and so there are huge opportunities here. Identify a niche ("fast accurate and timely web metrics for decision-making"), a tool that can solve it (MongoDB), and build the deployment, scaling, administration, reporting tools so you can sell a complete package into that niche. Rinse, lather, repeat.

The War For the Web

by Tim O'Reilly | @timoreilly | comments: 61

On Friday, my latest tweet was automatically posted to my Facebook news feed, as always. But this time, Tom Scoville noticed a difference: the link in the posting was no longer active.

It turns out that a lot of other people had noticed this too. Mashable wrote about the problem on Saturday morning: Facebook Unlinks Your Twitter Links.

if you’re posting web links (Bit.ly, TinyURL) to your Twitter feed and using the Twitter Facebook app to share those updates on Facebook too, none of those links are hyperlinked. Your friends will need to copy and paste the links into a browser to make them work.

If this is a design decision on Facebook’s part, it’s an extremely odd one: we’d like to think it’s an inconvenient bug, and we have a mail in to Facebook to check. Suffice to say, the issue is site-wide: it’s not just you.

As it turns out, it wasn't just links imported from Twitter. All outbound links were temporarily disabled, unless users explicitly added them as links via an "attach" dialogue. I went to Facebook, and tried posting a link to this blog directly in my status feed, and saw the same behavior: links were no longer automatically made clickable. You can see that in the image that is the destination of the first link in this piece.

The problem was quickly fixed, with URLs in status updates automatically now linkified again. The consensus was that it was in fact a bug, but it's little surprise that people suspected otherwise, given the increasing amount of effort Facebook puts into warning people that they are leaving Facebook for the big bad unsafe Internet:

All of this is well-intentioned, I'm sure. After all, Facebook is attempting to put in place privacy controls that allow its users to manage the visibility of their information -- and the Web's expectation of universal visibility is not necessarily the best default for much of the information posted on Facebook. But let's not kid ourselves: Facebook is a new kind of web site (or an old kind redux), a world of its own, playing by different rules.

But this isn't just about Facebook.

The Apple iPhone is the hottest web access device around, and like Facebook, while it connects to the web, it plays by a different set of rules. Anyone can put up a website, or launch a new Windows or Mac OS X or Linux application, without anyone's permission. But put an app onto the iPhone? That requires Apple's blessing.

There is one glaring loophole: anyone can create a web application, which any user can save as clickable application on their phone. But these web applications have limits - there are key capabilities of the phone that are not accessible to web applications. HTML 5 can introduce all the new application-like features it wants, but they will work only for web applications, and can't access key aspects of the phone with Apple's permission. And as we saw earlier this year with Apple's rejection of the Google Voice application, Apple isn't shy about blocking applications that it considers threatening to their core business, or that of their partners.

And now, of course, we see the latest salvo in the war against the accepted rules of interoperability on the web: Rupert Murdoch's threat to take the Wall Street Journal out of the Google search index. While most people have repeated the existing wisdom that to do so would be suicide for the Journal, a few contrarian observers have noted the leverage Murdoch holds. Mark Cuban argues that Twitter now trumps search engines when it comes to breaking news. Even more provocatively, Jason Calacanis suggested, a few weeks before Murdoch's announcement, that all big media companies need to do to cut Google off at the knees would be to block Google, while cutting an exclusive deal with Bing to be found only in Microsoft's search index.

Of course, Google wouldn't take that lying down, and would likely make its own exclusive deals, leading to a showdown that would make the browser wars of the 90s seem tame.

I'm not saying that News Corp and other mainstream media publications would adopt Jason's suggested strategy, or that it would work if they did, but it is becoming clear to me that we are heading into a bloody period of competition that could be extremely unfriendly to the interoperable web as we know it today.

If you've followed my thinking about Web 2.0 from the beginning, you know that I believe we are engaged in a long term project to build an internet operating system. (Check out the program for the first O'Reilly Emerging Technology Conference in 2002 (pdf).) In my talks over the years, I've argued that there are two models of operating system, which I have characterized as "One Ring to Rule Them All" and "Small Pieces Loosely Joined," with the latter represented by a routing map of the Internet.

The first is the winner-takes-all world that we saw with Microsoft Windows on the PC, a world that promises simplicity and ease of use, but ends up diminishing user and developer choice as the operating system provider.

The second is an operating system that works like the Internet itself, like the web, and like open source operating systems like Linux: a world that is admittedly less polished, less controlled, but one that is profoundly generative of new innovations because anyone can bring new ideas to the market without having to ask permission of anyone.

I've outlined a few of the ways that big players like Facebook, Apple, and News Corp are potentially breaking the "small pieces loosely joined" model of the Internet. But perhaps most threatening of all are the natural monopolies created by Web 2.0 network effects.

One of the points I've made repeatedly about Web 2.0 is that it is the design of systems that get better the more people use them, and that over time, such systems have a natural tendency towards monopoly.

And so we've grown used to a world with one dominant search engine, one dominant online encyclopedia, one dominant online retailer, one dominant auction site, one dominant online classified site, and we've been readying ourselves for one dominant social network.

But what happens when a company with one of these natural monopolies uses it to gain dominance in other, adjacent areas? I've been watching with a mixture of admiration and alarm as Google has taken their dominance in search and used it to take control of other, adjacent data-driven applications. I noted this first with speech recognition, but it's had the biggest business impact so far in location-based services.

A few weeks ago, Google offered free turn-by-turn directions for Android phones. This is awesome news for consumers, who previously could get this only in dedicated GPS devices or with high-priced iPhone apps. But it's also a sign just how competitive the web is getting, and just how powerful Google is getting, because they understand that "data is the Intel Inside" of the next generation of computer applications.

Nokia paid $8 billion for NavTeq, the leading provider of such turn-by-turn directions. GPS-maker TomTom paid $3.7 billion for TeleAtlas, the #2 provider in the market. Google quietly built an equivalent service, and is now giving it away for free -- but only to their own business partners. Everyone else still has to pay high fees to NavTeq and TeleAtlas. What's more, Google upped the ante by adding in such features as Street View.

Most interestingly, this move sets the stage for the future competition between Google and Apple. (Bill Gurley's analysis is an essential read.) Apple controls access to the dominant device of the mobile web; Google controls access to one of the most important mobile applications, and so far, is making it available for free only on Android. Google's prowess is not just in search, but in mapping, speech recognition, automated translation, and other applications driven by huge, intelligent databases that only a few providers can offer. Microsoft and Nokia control comparable assets, but they too are Apple competitors, and unlike Google, their business model depends on selling access to those assets, not giving them away for free.

It could be that everyone will figure out how to play nicely with each other, and we'll see a continuation of the interoperable web model we've enjoyed for the past two decades. But I'm betting that things are going to get ugly. We're heading into a war for control of the web. And in the end, it's more than that, it's a war against the web as an interoperable platform. Instead, we're facing the prospect of Facebook as the platform, Apple as the platform, Google as the platform, Amazon as the platform, where big companies slug it out until one is king of the hill.

And it's time for developers to take a stand. If you don't want a repeat of the PC era, place your bets now on open systems. Don't wait till it's too late.

P.S. One prediction: Microsoft will emerge as a champion of the open web platform, supporting interoperable web services from many independent players, much as IBM emerged as the leading enterprise backer of Linux.

I'll be speaking on this topic in my keynote at the Web 2.0 Expo in New York on Tuesday. I'll look forward to seeing many of you there.

Four short links: 16 November 2009

Visualizing Adventures, Droid Deployments, Fly Vision, and Mass Meat For You

by Nat Torkington | @gnat | comments: 0

1. Choose Your Own Adventure -- numerical and visual analysis of the Choose Your Own Adventure novels. The distinguishing characteristic of My Kind Of People is that they appreciate the quantitative study of the commonplace. (via Bryan O'Sullivan)
2. Tracking Droid Numbers -- uLocate, the makers of the Where app for Android, have been tracking the growth of the Droid phone using the data they get from the Android app store. (via BoyGenius Report)
3. Fly Eyes Makes Better Robot Vision -- to make smaller flying robots, researchers would like to find a simpler way of processing motion. Inspiration has come from the lowly fly, which uses just a relative handful of neurons to maneuver with extraordinary dexterity. And for more than a decade, O’Carroll and other researchers researchers have painstakingly studied the optical flight circuits of flies, measuring their cell-by-cell activity and turning evolution’s solutions into a set of computational principles. [...] Intriguingly, the algorithm doesn’t work nearly as well if any one operation is omitted. The sum is greater than the whole, and O’Carroll and Brinkworth don’t know why. Because the parameters are in constant feedback-driven flux, it produces a cascade of non-linear equations that are difficult to untangle in retrospect, and almost impossible to predict. (via Slashdot)
4. Meat Band Aids and Mass Production of Living Tissue -- Apligraf is a matrix of cow collagen, human fibroblasts and keratinocyte stem cells (from discarded circumcisions), that, when applied to chronic wounds (particularly nasty problems like diabetic sores), can seed healing and regeneration. This Gizmodo Q&A is informative.

Recent Posts

• Ignite NYC on 11/16: Gov 2.0, Body Hacks, and Hi-Tech Craft | by Brady Forrest on November 15, 2009
• It's in the Bag! The Apple Tablet Computing Device | by Mark Sigal on November 13, 2009
• Four short links: 13 November 2009 | by Nat Torkington on November 13, 2009
• Four short links: 12 November 2009 | by Nat Torkington on November 12, 2009
• Quarantined Conferences: Claustrophobic Technophiles or Attentive Audiences? | by Mark Drapeau on November 11, 2009
• Counting Unique Users in Real-time with Streaming Databases | by Ben Lorica on November 11, 2009
• Four short links: 11 November 2009 | by Nat Torkington on November 11, 2009
• Converting to Electronic Health Records: fits and starts | by Andy Oram on November 10, 2009
• Four short links: 10 November 2009 | by Nat Torkington on November 10, 2009
• The Minds Behind Some of the Most Addictive Games Around | by James Turner on November 9, 2009
• Four short links: 9 November 2009 | by Nat Torkington on November 9, 2009
• Unlikely Group Working Happily Together To Solve Patent Problem | by Carl Malamud on November 8, 2009