Time passes (months, maybe years) without your noticing. One day, you’re sitting in a café reading the news on your phone and you notice that the person next to you has some really nice footwear. No QR code on the neighbor’s shoes? No problem. You start up your visual search application (no, it’s not available on all handsets), act like you are trying to find something on the newsreader screen while you turn off the camera’s “snap the photo” sound, then discreetly aim and take the photo of the shoes. No one has noticed, right? You wait, you act like you’re still reading the news. Your mobile browser opens and on your screen is the exact model of shoes on your neighbor’s feet. Another click and you can check the price and availability from stores nearby.

The world is an interactive catalog

The first Sears catalog was published in 1888 for farmers who came to town only rarely and needed to place orders with someone other than their local merchant. By 1894, the Sears catalog had grown to 322 pages, featuring sewing machines, bicycles, and even cars. The next year it exceeded 500 pages and contained more products than people had previously thought existed. About 100 years later, the World Wide Web was getting off the ground, and in the past 15 years we’ve witnessed the explosion of information and had our eyes opened to a vast universe of merchandise, including customized versions of many items made on demand for our pleasure.

Augmented Reality shopping

Much like the farm wife with her Sears catalog, consumers will be able to use simple AR applications to make more informed buying decisions.

Items that are fixed in place (a building, for example) are well suited to research and information display using GPS and compass-based mobile AR as we know it today. Shopping for a new house or apartment to buy or rent? Among the original 4 or 5 layers in the mobile AR application Layar, is a real estate application. Via the Layar partner Funda, Layar launched with a few dozen properties. Now, with 17 real estate partners, tens of thousands of properties are listed worldwide. Perhaps, with just a few more lines of code, could the entire US Multiple Listing Service database be accessible from Layar or a similar application?

AR shopping for objects which move or in indoor environments (where GPS doesn’t work as well, if at all) is a little more challenging. But merchants and brands are always looking for novel ways to enhance a retail experience, and AR technology is on the list of tools that could serve the customer. In 2006, Michigan State University researchers supported by a Microsoft Research grant designed the PromoPad system, a shopping assistant based on a TabletPC which would ride in the shopping cart “cradle.” Since image recognition was not sufficiently advanced to serve as a way to identify objects (and this was not the focus of their study), a combination of fiducials (another name for 2D bar codes) and RFID were used to detect the objects around the PromoPad user. The team put together a user interface for simple browsing, thought about how to attractively overlay information on the real world using a video camera connected to the TabletPC and how to design the shopping environment to make sure that only the items most likely to be of interest were presented to the shopper.

Using similar concepts, Insqribe is offering what appears to be a commercial version the PromoPad. They call it a real-time proximity-based marketing system. One version of Insqribe’s system still requires a marker or 2D barcode for recognition. They have also implemented a version with AR features. Another example of 2D barcode-based AR shopping is provided by Moving Brands. The current version of Moving Brands does not use the natural features of an object to recognize the product which has the attention of the shopper.

Image recognition as the basis for obtaining information in an AR-enhanced retail application is here today, but only for a few classes of products. Books, CDs and DVDs are the low hanging fruit for this application because the image databases for these searches are already on-line and the “planar” objects (flat) are far easier to recognize than 3D objects. One of the iPhone applications for the books, CDs and DVD search application is provided free by Swiss start-up kooaba. In June 2009, Amazon’s A9.com division acquired another company leading in this domain, SnapTell. Although I haven’t tried it personally, I’m told that the technology is already in Amazon’s service. [Note: please don’t confuse the technology which this post is about with Microsoft’s Bing Visual Search. Same term, Bing is a close “cousin” of image search for retail.]

In the case of books, CDs and DVDs, you might be asking yourself why you would need this application if the real object (e.g., the book about which you want more information) is already in your hand. Some of the answer lies in formats. Maybe you would like to purchase and read the digital version of the book you’re holding. Maybe you would like to listen to the music on your iPod but you are holding the CD. Perhaps you would like to see the movie you are holding in DVD format on a large screen in a theater in your neighborhood.

Good news or bad news?

Like the Sears catalog over a century ago, AR with visual search will cause the minds of people around world to open to entirely new possibilities. Sounds promising? For many people, having the world around them in the form of a searchable, interactive catalog is a distasteful, repugnant reminder that everything is for sale in a hyper-materialistic society. For others, it will become an addiction, perhaps causing people to buy beyond their means.

There is also the bright side of turning the world into an interactive catalog. Motivations for using AR for shopping could include the need or passion to be a frugal shopper. The applications will be able to support price comparisons and, using GPS, perhaps recommend a shop for immediate purchase. Too busy to go into a store to browse? Some will embrace mobile catalog shopping for the same reason they prefer on-line shopping: busy lifestyle. There are also those consumers who have a keen desire to adopt more careful, let’s say “informed,” habits with respect to their consumption of physical goods and services. They want to purchase and consume materials which they know to be wholesome, damage the planet as little as possible, and may even achieve some social good. An interactive catalog based on images could alleviate the need to get a bar code or to have the exact model number of a product.

And, adding this new functionality will permit companies to explore new business models. Would consumers be willing to pay a premium for a faster or higher-accuracy visual search engine for retail? Would consumers be willing to pay a premium to have reminders sent to them when they are near a retail outlet where a previously “spotted” object is available in their price range?

Try it on for size and style

AR is also useful when combined with a PC or another device in a virtual dressing room application. For example, Zugara is proposing to develop for its clients customized versions of a Webcam Social Shopper application which allows prospective customers to virtually try on clothes and use motion (actually gesture) detection algorithms for keyboard- and mouse-free navigation (another use for computer vision). Cisco has just released a short commercial vision video which shows how such technology will help shoppers in the future.

This virtual catalog combined with a dressing room concept is attractive but there remain some questions. What happens if the person using the application is a different size than the model on which the garment was originally cast? Are there different markers for people with different height and weight? Then the key question is if this indeed increases retail sales as Zugara suggests.

Feet back on the ground

For many of the location-based shopping services available today, such as recommendations of restaurants and places to hang out, we already have examples of the use of AR (see for example, the Monocle feature of the Yelp iPhone application and, in Japan, Tonchidot‘s Sekai Camera). We should see AR integration as a feature of more navigation and recommendation services in the future.

Although image recognition technologies for AR applications on mobile handsets are not quite ready to take on the challenge of identifying the pair of shoes your neighbor is wearing, they are entirely up to the task of identifying, from a well-lit photograph, the model of a car or a logo. Expect retail outlets and brands that provide fast-moving consumer goods to be among those eager to exploit mobile AR for shopping.

Shopping with AR is a shoe-in for some, but not all things we might want or need. There remain gaps in this ecosystem which will take a few years to fill. For example, the consumer may wish to consummate a purchase directly from the application which has an AR feature. Mobile commerce is far from a fait acompli. Transaction systems must be integrated to the interactive catalog platforms.

So, despite a high potential for both merchants and consumers to use AR for shopping, we must not get too far ahead of the cart with this one. What do you think? Is shopping with AR something you would do? When do you think it will be ready for prime time?

Researchers at the Max Planck Institute for Biological Cybernetics at Tuebingen and the Swiss Federal Polytechnical Institute in Zurich (ETHZ) are members of a worldwide community conducting studies with users to evaluate how touch and sight work together in multimodal interfaces for AR applications. These researchers are making great strides in the direction of “next generation” user interfaces, inventing devices that take advantage of increasingly powerful and sensitive sensors (one and then two cameras, Assisted GPS, 3D magnetometer, or 3D accelerometer).

Touch

In most consumer mobile AR applications released to date, AR interaction is mediated via haptics–the user touches a highlighted area of the screen to request to more information associated with an object or point on the planet.

But we perceive our environment using a combination of all senses, and researchers are developing haptic interfaces that go beyond the “touch for more info” model. Dr. Matthias Harders of the ETHZ Computer Vision Lab works on applications of AR for training surgeons. At the upcoming ISMAR 09 meeting, Drs. Benjamin Knörlein (ETHZ), Massimiliano Di Luca (Max Planck), and Harders will present the results of their ongoing research on the use of haptics and AR interfaces. Their studies show that delays for haptic feedback result in decreases in the user’s perception of “stiffness” in the interface. In contrast, visual delays (where there is a delay between the touch and the response) caused an increase in perceived stiffness. Understanding how vision and touch interact to affect the user’s perception can help AR developers finetune the interface so that it accurately maps that perception.

Such studies may, one day, be helpful for designing AR training for neurosurgeons so, please, pay attention!

Show Me

Today, the information associated with a Point of Interest (POI) in a consumer AR application is frequently presented on the small screen as either text in a box or bubble, or arrows for navigation. These are easy to understand and “good enough,” given the low level of accuracy and speed in today’s sensors. Their results are fine for a tourist trying to locate a subway stop, but would not be suitable for a utility crew registering the precise position of an underground electrical cable before digging.

One drawback of overlaying text or diagrams on a live video showing on a mobile phone screen is that the user must hold the device in the correct position, pointing at the target. Head-Mounted Displays (HMDs) offer a hands-free, heads-up alternative. For at least a decade, researchers in dozens of laboratories and companies have worked on the development of lighter and more reliable HMD technologies. Many research systems, systems designed for professional training applications, and military applications (e.g., night vision goggles) use HMDs.

While popular consumer applications which use HMDs are likely to be more than three years in the future, the cost is falling, performance is rising, and more applications for HMDs exist today than many realize. For example, the Academy of Art and Design at the University of Applied Sciences Northwestern Switzerland has developed a system called LifeClipper2 which uses a heads-up display that allows the user to experience the changes proposed for urban renewal or development projects. LifeClipper can help city officials and other stakeholders more fully understand the impact of those projects before they’re actually built.

Many mobile phone AR applications require the user to hold up a mobile handset for several minutes, and that’s tiring. But avoiding that fatigue isn’t the only compelling reason to explore the future of HMDs. [Note: Thanks to V-VM of Nokia Research Center for reminding me that there may be camera phones designed with the lens and screen positioned in such a way that the user would not need to hold up their arm to view the world at eye level. But then the user would be looking down, which might also be awkward when walking.]

HMDs have three other noteworthy advantages. First, in bright light conditions when a mobile-phone screen may be difficult to see, these displays remain very visible for the user. The heads-up display can even become the user’s sunglasses. Don’t you think I look chic in these shades?

Christine Perey wearing last year’s model of the Nokia gaze tracking system [photo credit: Toni Järvenpää]

Second, when the display is resting on the user’s ears and nose, as eyewear, their hands are free to do something else. If, however, the display is near the eye, the question of how the user selects the point of interest becomes more problematic. Scientists at the Nokia Research Center in Finland have been working on using eye gaze to point. Gaze detection and head position tracking for directing user interactions are the purpose of the device worn in the photo to the left.

Finally, although the field of view is still much less wide than that required for many AR and VR applications, an HMD can provide a more “immersive” user experience than a small handheld screen at arms length. Frequently, HMDs also have integrated ear pieces to augment the visual with sound. Using sounds (for example, 3D audio) and synthesized or natural speech for presenting information to the user is yet another area of exploration which is sure to be underway in research labs around the world.

To appreciate the full potential of new user interaction paradigms being developed for and studied with AR, a test drive is valuable. At ISMAR 09 in Orlando October 20-22, 2009, Vuzix, a manufacturer of a wide range of HMDs will be among several in the business exhibiting their latest models. As part of the research demonstrations, Nokia Research will be showing the current prototype of its gaze tracking, using hardware which is much lighter than that which was used a year earlier (shown above). A YouTube video showing how the new model might work in the future can be watched here. Microvision, another manufacturer of commercial HMDs and pico-projectors, will be participating in one of the ISMAR Workshops on October 19, 2009.

Rather than focusing entirely on the window itself, it’s time to look at the value the information in the window is bringing to the user.

Let’s study the user’s interaction with digital information about their world on a deeper level. What are people likely to need AR applications to do for them? The hottest applications for AR in the next year will closely resemble familiar human interactions with the physical world.

High Level Taxonomy

In order to really understand the requirements of AR applications, we need clearly defined application categories.

One stab at an AR application taxonomy distinguishes between those applications which require a very highly controlled, perhaps a highly “instrumented” user environment, and those which do not. Intersense is one of the leading providers of technologies for AR applications that require knowing the precise position of user and objects in a reference framework. Examples of highly-instrumented environments are a technology-assisted operating theater in a hospital, a virtual assembly line in a manufacturing plant, an oil drilling platform, a museum with AR exhibits or the space in which visitors can experience an AR-enhanced ride within a theme (amusement) park.

Then there are uncontrolled environments: anywhere that lacks unique identifiers placed in advance for the benefit of the AR user’s application, and where the environmental conditions may rapidly change.

As I was preparing the description of the upcoming Mobile Magic Wand workshop, I discovered that there also needed to be a distinction made between what I call “mobile AR” and that which the chairs of the Let’s Go Out workshop, a workshop conducted in parallel with the Mobile Magic Wand meeting, call simply “outdoor AR.”

Here’s what we came up with: Mobile AR applications or services can use an off-the-shelf hardware platform, such as a mobile phone, UMPC or personal digital assistant with pre-integrated sensors (GPS or camera, for example), or a custom-designed system of any specification as long as the user can carry it without assistance and the device is not connected to its database by a physical cable of fixed length. The access to digital information can be mediated by a wireless or cellular network connection or the digital information necessary for the AR application can all be kept local to the user’s device (pre-loaded).

Mobile AR can be both indoor and outdoor. The scope of “Outdoor AR” includes that portion of mobile AR for which the use case is outside of any building or shelter. There may be some Outdoor AR applications which are not highly portable. Mobile AR includes only that portion of Outdoor AR which is accessible with a device a user can carry without assistance, which is not connected to a server by means of a physical cable and is entirely outside a highly-instrumented environment.

Here’s a photo of me wearing the LifeClipper2 system.

Christine Perey using lifeClipper2 in Basel [photo credit: Jan Torpus]

It’s mobile, at least I could carry it unassisted, and I’m outdoors.

Mobile AR applications

Let’s explore the next level in the Mobile AR application taxonomy. There are professional applications for Mobile AR: this covers all the uses for a mobile AR system by people doing their job.

In one of the LifeClipper2 scenarios, for example, urban planners wear the system to experience (during the design phase of a project) the changes they propose to make in the real environment. These changes could include introducing new vegetation, removal of existing buildings and, perhaps, addition of new facilities. The mixture of simulated objects with the real world is very fluid and, in some instances, even includes changing the acoustic properties in the space.

In the class of Mobile AR for consumers, there are navigation applications or services. Primarily pedestrian navigation, these way-finding applications help us get to the nearest bus stop, public rest rooms, ATM machine, a doctor’s office, coffee shop, McDonald’s and other fixed point which we might need to find in geo-space. We could find these points the old-fashioned way, by asking someone or using a map, but is easier for some to navigate by means of a screen and using the phone’s GPS and compass than on a digital or paper map view.

The next category of Mobile AR consumer applications is associated with having fun: games which involve the user’s natural environment as stimuli or just weave together the natural world and synthetic game objects and players. Think of the Kweekies game, or the AcrossAir virus killer 360 application. In this rich and expanding category, the user’s interaction with the surroundings by way of a consumer mobile telephone or PDA is playful, or at the very least involves earning points in some way. Frequently the metaphor is pointing and shooting or tapping. I’m not implying that these applications aren’t purposeful because having fun is a widely-felt human need, but it is distinctly different from other application categories.

Social AR is the third category which is ripe with possibilities. It is at the intersection of social networks and AR interfaces. Social AR applications will seek to fulfill the human need to find people and to share with friends and fellow inhabitants something personal about ourselves. Social AR will permit us to annotate the places we co-occupy or which we have occupied at different times. Some Social AR applications will arise from location-based social network services such as Yelp’s Monocle feature in its iPhone application.

Mobilizy has made a big contribution in this category with the release of Wikitude.me. Expect there to be many other examples of social AR coming out in the near future. Social AR will undoubtedly also have aspects in common with game applications when two or more people are using their AR interfaces to play with one another.

There are more AR application categories. Can you suggest a few? What are your favorite Mobile AR application segments?

Since most of the world’s people, objects and places are not emitting radio signals which our mobile Internet devices can reliably detect, as was once envisioned in the early visions of RFID, other technologies are being used and new ones being developed for detection in AR applications. Further, even if there were tags on us (or other moving objects) and readers everywhere, RFID alone is insufficient to provide the six degrees of freedom necessary to correctly position a device relative to the object or point of interest. This isn’t to say that RFID has no place at all in AR, just that it is not a widely applicable tool for developers of today’s consumer AR applications.

Tracking for AR applications involves identification of one or more targets in the user’s field of vision or surroundings, then keeping track of the position of the user’s device relative to the recognized and/or selected object in three-dimensional space, and, for there to be an augmentation in the field of view, properly “registering” an overlay image or text to the real world object. The first two of these steps are closely aligned with the sequence which some types of robots need to perform when moving autonomously in an environment. They are also leveraging core ubiquitous computing technologies which are necessary in “intelligent environments,” as in spaces which exhibit Ambient Intelligence.

Tracking real world objects which are stationary (with fixed geo-location coordinates) has been achieved most widely and at relatively low cost using a mobile phone’s GPS and compass. There are many examples such as Wikitude, Layar and BionicEye. But there are situations in which GPS and compass are not the best, for example when the user is inside a building or near something which causes disturbances in the magnetic field and, in the best of circumstances, GPS and compass technology don’t provide the speed and accuracy which many AR applications require.

Let’s just take, for example, applications in which the user’s target object is not fixed in space. This challenge has been solved for years by affixing a marker, such as a QR (Quick Response) or Data matrix code on the object. For the past three or four years, markers have provided a suitable approximation for what most people designing AR applications really want: recognition of people, objects or places on the basis of their unique features, or, in the research community vernacular, “natural feature recognition.”

Not surprisingly, tracking with feature recognition technology is currently receiving a great deal of attention in the scientific community and will be a field of future research among user experience designers, computer vision experts and a variety of other domains for some time to come. Scientists from world-renown centers of research on tracking are going to be presenting a few of their most recent achievements at the annual conference of the Mixed and Augmented Reality research community later this month in Orlando.

In his paper, to be published in the conference proceedings, Vincent Lepetit, a researcher at the Computer Vision Lab at EPFL will be describing how his group has extended the ESM (Efficient Second-order approximation Method) algorithm developed by researchers at INRIA in order to compensate for motion blur when tracking a 3D object. Another achievement of the CVLab in Lausanne is the ability to identify (to detect) a 3D object’s features without computationally and time-intensive prior training.

Also in attendance at ISMAR, the Institute for Computer Graphics and Vision at the Technical University of Graz (Austria) has a group of researchers focusing specifically on algorithms for natural feature recognition of objects for AR applications on mobile phones and has broken a number of significant barriers in this field, notably the Studierstube Tracker, a library for 2D code detection on the mobile handset for AR applications. Currently in its second phase, the Handheld AR project is leveraging past work at TU Graz and in other labs, in the field of natural feature recognition.

In their ISMAR 2009 paper, Daniel Wagner, Dieter Schmalstieg and Horst Bischof of TU Graz describe a new technique for very high accuracy and real-time pose estimation and tracking on mobile phones. Another of the many interesting projects on which the institute’s faculty and students are working is the development of technology permitting utility workers to use their mobile handsets for “X-ray vision” of underground structures while in the field.

RFID remains an interesting option to supplement other tracking technologies for indoor applications and situations which are relatively tightly controlled (e.g., teaching/training, museums, entertainment venues, architecture and urban planning). When RFID readers are more ubiquitous, as common as GPS and compass, for example, the technology could play a role in future AR applications to narrow down the field of search and reduce the time required for a positive identification, for example. Tracking for consumer AR applications in uncontrolled environments when all the user has is a camera phone remains a very, very challenging area of research and we should expect to continue seeing major developments in this field in the year ahead before it is gradually integrated into our everyday AR applications.