Look through the AR window

“Augmented Reality is your window to the world,” suggest some companies who have released applications for the iPhone 3G S and Android handsets over the past few months. As we have witnessed since the launch of Layar in June, the world needed another “window.” Or, perhaps we needed another browser? Have you heard about the World Surfer by GeoVector?

Rather than focusing entirely on the window itself, it’s time to look at the value the information in the window is bringing to the user.

Let’s study the user’s interaction with digital information about their world on a deeper level. What are people likely to need AR applications to do for them? The hottest applications for AR in the next year will closely resemble familiar human interactions with the physical world.

High Level Taxonomy

In order to really understand the requirements of AR applications, we need clearly defined application categories.

One stab at an AR application taxonomy distinguishes between those applications which require a very highly controlled, perhaps a highly “instrumented” user environment, and those which do not. Intersense is one of the leading providers of technologies for AR applications that require knowing the precise position of user and objects in a reference framework. Examples of highly-instrumented environments are a technology-assisted operating theater in a hospital, a virtual assembly line in a manufacturing plant, an oil drilling platform, a museum with AR exhibits or the space in which visitors can experience an AR-enhanced ride within a theme (amusement) park.

Then there are uncontrolled environments: anywhere that lacks unique identifiers placed in advance for the benefit of the AR user’s application, and where the environmental conditions may rapidly change.

As I was preparing the description of the upcoming Mobile Magic Wand workshop, I discovered that there also needed to be a distinction made between what I call “mobile AR” and that which the chairs of the Let’s Go Out workshop, a workshop conducted in parallel with the Mobile Magic Wand meeting, call simply “outdoor AR.”

Here’s what we came up with: Mobile AR applications or services can use an off-the-shelf hardware platform, such as a mobile phone, UMPC or personal digital assistant with pre-integrated sensors (GPS or camera, for example), or a custom-designed system of any specification as long as the user can carry it without assistance and the device is not connected to its database by a physical cable of fixed length. The access to digital information can be mediated by a wireless or cellular network connection or the digital information necessary for the AR application can all be kept local to the user’s device (pre-loaded).

Mobile AR can be both indoor and outdoor. The scope of “Outdoor AR” includes that portion of mobile AR for which the use case is outside of any building or shelter. There may be some Outdoor AR applications which are not highly portable. Mobile AR includes only that portion of Outdoor AR which is accessible with a device a user can carry without assistance, which is not connected to a server by means of a physical cable and is entirely outside a highly-instrumented environment.

Here’s a photo of me wearing the LifeClipper2 system.

Christine Perey using lifeClipper2 in Basel [photo credit: Jan Torpus]

It’s mobile, at least I could carry it unassisted, and I’m outdoors.

Mobile AR applications

Let’s explore the next level in the Mobile AR application taxonomy. There are professional applications for Mobile AR: this covers all the uses for a mobile AR system by people doing their job.

In one of the LifeClipper2 scenarios, for example, urban planners wear the system to experience (during the design phase of a project) the changes they propose to make in the real environment. These changes could include introducing new vegetation, removal of existing buildings and, perhaps, addition of new facilities. The mixture of simulated objects with the real world is very fluid and, in some instances, even includes changing the acoustic properties in the space.

In the class of Mobile AR for consumers, there are navigation applications or services. Primarily pedestrian navigation, these way-finding applications help us get to the nearest bus stop, public rest rooms, ATM machine, a doctor’s office, coffee shop, McDonald’s and other fixed point which we might need to find in geo-space. We could find these points the old-fashioned way, by asking someone or using a map, but is easier for some to navigate by means of a screen and using the phone’s GPS and compass than on a digital or paper map view.

The next category of Mobile AR consumer applications is associated with having fun: games which involve the user’s natural environment as stimuli or just weave together the natural world and synthetic game objects and players. Think of the Kweekies game, or the AcrossAir virus killer 360 application. In this rich and expanding category, the user’s interaction with the surroundings by way of a consumer mobile telephone or PDA is playful, or at the very least involves earning points in some way. Frequently the metaphor is pointing and shooting or tapping. I’m not implying that these applications aren’t purposeful because having fun is a widely-felt human need, but it is distinctly different from other application categories.

Social AR is the third category which is ripe with possibilities. It is at the intersection of social networks and AR interfaces. Social AR applications will seek to fulfill the human need to find people and to share with friends and fellow inhabitants something personal about ourselves. Social AR will permit us to annotate the places we co-occupy or which we have occupied at different times. Some Social AR applications will arise from location-based social network services such as Yelp’s Monocle feature in its iPhone application.

Mobilizy has made a big contribution in this category with the release of Wikitude.me. Expect there to be many other examples of social AR coming out in the near future. Social AR will undoubtedly also have aspects in common with game applications when two or more people are using their AR interfaces to play with one another.

There are more AR application categories. Can you suggest a few? What are your favorite Mobile AR application segments?

  • Hi Christine,
    Thanks for the mention in the piece : )
    GeoVector’s app is called “World Surfer”
    It’s available for Android & iPhone.
    Peter Ellenby

  • Hi Christine,

    There will be so many categories. What taxonomy do we use for webpages?

    The longtail of mobile context will make it hard to find a taxonomy that will hold.

    Also expect categories to emerge that we have not seen in other media yet. Categories that are native to the AR medium but unknown to other media.

    I even expect the best AR apps to have social, functional and game elements.. Does Foursquare have all three of these? ;-)

  • Hi Raimo, I agree with you that there will be many, many applications and that people will invent new, exciting uses for AR when they have experimented more. It is unlikely that a framework proposed today will hold for all time!

    My suggestion for the near term, let’s say for the next 12 months while the uses closely resemble what we do in the real world without AR, is that we have a taxonomy which permits a discussion between partners about specific requirements and user expectations.

    What would you call Layar? Is it “a browser” that it can be used to interact with content for many applications? Is it an engine for linking hardware and software components in the handset with content in the cloud? Is it a platform for publishing AR applications?


  • We like the web browser analogy. We believe that AR experiences can be captured within one browser. Not a web browser but an AR browser. Next to being a generic AR viewer we ofcourse also enable content discovery through for example search and offer developers the possibility to design their own AR experiences (games, functional search layers or rich experiences) and publish these in our browser.

    Would not know in which box you could fit us ;-)

  • Augmented reality also can be implemented in business and business relations. People, such as me (http://visualcard.me), are working in embedding augmented reality in business cards.

    Carefully done, augmented reality can be introduce to business man without the need of all the fancy stuff.

  • I would add Event AR as a category within Mobile AR for consumers. The IBM Seer project for Wimbledon is interesting in providing localized navigation (to courts, washrooms, VIP lounge…), real time data in terms of taxi queues, twitter.. Would envision that this could blend traditional event program information with local navigation and real-time data/news, and provide a new type of information/experience offering for events, or in some cases places.

  • We recently launched a real-time marketing platform for AR, see http://insqribe.com/augmented-reality. We allow brands and retailers to jump on board with AR right now, without the need for any developers.
    All interactions on our platform are measured and tracked so you can watch people interact with your AR experience in real-time.

    We are disappointed that we cannot attend ISMAR 09 but will be in the US very shortly!


  • Hi Danny, Are you suggesting that Shopping is another category of applications for Mobile AR?

    I’m quite keen to see this in real world.

    There are some experiments and proof of concepts which have been done but many missing pieces left to be designed.

    I think the proximity piece of your platform is important and it should be made clear to people that the system you are proposing does not rely on GPS but does require a 2D barcode reader and for the packaging to have a marker/fiducial already on it in order for the system to work.


  • Hi Christine,

    We actually offer two solutions on our proximity marketing platform. 2D Barcodes are one method of engaging consumers, but we have also released an AR platform. Both 2D Barcodes and AR work very well in parallel.