Shopping with AR

You are sitting in a café reading the newspaper and there’s a QR code on the page near an article which you found interesting. You’ve never seen this box symbol in a newspaper before but, since you already know the general purpose of these funny black and white checkered boxes, you ask yourself what this is about. You start up your QR code reader application (yes, it’s pre-installed on many camera phones already), aim, see the digital box appear on your screen around the QR code and snap a picture. Off goes the code and, depending on your mobile network connection, somewhere between 30 seconds and 2 minutes later more details about this feature story appear in your mobile browser screen.

Time passes (months, maybe years) without your noticing. One day, you’re sitting in a café reading the news on your phone and you notice that the person next to you has some really nice footwear. No QR code on the neighbor’s shoes? No problem. You start up your visual search application (no, it’s not available on all handsets), act like you are trying to find something on the newsreader screen while you turn off the camera’s “snap the photo” sound, then discreetly aim and take the photo of the shoes. No one has noticed, right? You wait, you act like you’re still reading the news. Your mobile browser opens and on your screen is the exact model of shoes on your neighbor’s feet. Another click and you can check the price and availability from stores nearby.

The world is an interactive catalog

The first Sears catalog was published in 1888 for farmers who came to town only rarely and needed to place orders with someone other than their local merchant. By 1894, the Sears catalog had grown to 322 pages, featuring sewing machines, bicycles, and even cars. The next year it exceeded 500 pages and contained more products than people had previously thought existed. About 100 years later, the World Wide Web was getting off the ground, and in the past 15 years we’ve witnessed the explosion of information and had our eyes opened to a vast universe of merchandise, including customized versions of many items made on demand for our pleasure.

Augmented Reality shopping

Much like the farm wife with her Sears catalog, consumers will be able to use simple AR applications to make more informed buying decisions.

Items that are fixed in place (a building, for example) are well suited to research and information display using GPS and compass-based mobile AR as we know it today. Shopping for a new house or apartment to buy or rent? Among the original 4 or 5 layers in the mobile AR application Layar, is a real estate application. Via the Layar partner Funda, Layar launched with a few dozen properties. Now, with 17 real estate partners, tens of thousands of properties are listed worldwide. Perhaps, with just a few more lines of code, could the entire US Multiple Listing Service database be accessible from Layar or a similar application?

AR shopping for objects which move or in indoor environments (where GPS doesn’t work as well, if at all) is a little more challenging. But merchants and brands are always looking for novel ways to enhance a retail experience, and AR technology is on the list of tools that could serve the customer. In 2006, Michigan State University researchers supported by a Microsoft Research grant designed the PromoPad system, a shopping assistant based on a TabletPC which would ride in the shopping cart “cradle.” Since image recognition was not sufficiently advanced to serve as a way to identify objects (and this was not the focus of their study), a combination of fiducials (another name for 2D bar codes) and RFID were used to detect the objects around the PromoPad user. The team put together a user interface for simple browsing, thought about how to attractively overlay information on the real world using a video camera connected to the TabletPC and how to design the shopping environment to make sure that only the items most likely to be of interest were presented to the shopper.

Using similar concepts, Insqribe is offering what appears to be a commercial version the PromoPad. They call it a real-time proximity-based marketing system. One version of Insqribe’s system still requires a marker or 2D barcode for recognition. They have also implemented a version with AR features. Another example of 2D barcode-based AR shopping is provided by Moving Brands. The current version of Moving Brands does not use the natural features of an object to recognize the product which has the attention of the shopper.

Image recognition as the basis for obtaining information in an AR-enhanced retail application is here today, but only for a few classes of products. Books, CDs and DVDs are the low hanging fruit for this application because the image databases for these searches are already on-line and the “planar” objects (flat) are far easier to recognize than 3D objects. One of the iPhone applications for the books, CDs and DVD search application is provided free by Swiss start-up kooaba. In June 2009, Amazon’s division acquired another company leading in this domain, SnapTell. Although I haven’t tried it personally, I’m told that the technology is already in Amazon’s service. [Note: please don’t confuse the technology which this post is about with Microsoft’s Bing Visual Search. Same term, Bing is a close “cousin” of image search for retail.]

In the case of books, CDs and DVDs, you might be asking yourself why you would need this application if the real object (e.g., the book about which you want more information) is already in your hand. Some of the answer lies in formats. Maybe you would like to purchase and read the digital version of the book you’re holding. Maybe you would like to listen to the music on your iPod but you are holding the CD. Perhaps you would like to see the movie you are holding in DVD format on a large screen in a theater in your neighborhood.

Good news or bad news?

Like the Sears catalog over a century ago, AR with visual search will cause the minds of people around world to open to entirely new possibilities. Sounds promising? For many people, having the world around them in the form of a searchable, interactive catalog is a distasteful, repugnant reminder that everything is for sale in a hyper-materialistic society. For others, it will become an addiction, perhaps causing people to buy beyond their means.

There is also the bright side of turning the world into an interactive catalog. Motivations for using AR for shopping could include the need or passion to be a frugal shopper. The applications will be able to support price comparisons and, using GPS, perhaps recommend a shop for immediate purchase. Too busy to go into a store to browse? Some will embrace mobile catalog shopping for the same reason they prefer on-line shopping: busy lifestyle. There are also those consumers who have a keen desire to adopt more careful, let’s say “informed,” habits with respect to their consumption of physical goods and services. They want to purchase and consume materials which they know to be wholesome, damage the planet as little as possible, and may even achieve some social good. An interactive catalog based on images could alleviate the need to get a bar code or to have the exact model number of a product.

And, adding this new functionality will permit companies to explore new business models. Would consumers be willing to pay a premium for a faster or higher-accuracy visual search engine for retail? Would consumers be willing to pay a premium to have reminders sent to them when they are near a retail outlet where a previously “spotted” object is available in their price range?

Try it on for size and style

AR is also useful when combined with a PC or another device in a virtual dressing room application. For example, Zugara is proposing to develop for its clients customized versions of a Webcam Social Shopper application which allows prospective customers to virtually try on clothes and use motion (actually gesture) detection algorithms for keyboard- and mouse-free navigation (another use for computer vision). Cisco has just released a short commercial vision video which shows how such technology will help shoppers in the future.

This virtual catalog combined with a dressing room concept is attractive but there remain some questions. What happens if the person using the application is a different size than the model on which the garment was originally cast? Are there different markers for people with different height and weight? Then the key question is if this indeed increases retail sales as Zugara suggests.

Feet back on the ground

For many of the location-based shopping services available today, such as recommendations of restaurants and places to hang out, we already have examples of the use of AR (see for example, the Monocle feature of the Yelp iPhone application and, in Japan, Tonchidot‘s Sekai Camera). We should see AR integration as a feature of more navigation and recommendation services in the future.

Although image recognition technologies for AR applications on mobile handsets are not quite ready to take on the challenge of identifying the pair of shoes your neighbor is wearing, they are entirely up to the task of identifying, from a well-lit photograph, the model of a car or a logo. Expect retail outlets and brands that provide fast-moving consumer goods to be among those eager to exploit mobile AR for shopping.

Shopping with AR is a shoe-in for some, but not all things we might want or need. There remain gaps in this ecosystem which will take a few years to fill. For example, the consumer may wish to consummate a purchase directly from the application which has an AR feature. Mobile commerce is far from a fait acompli. Transaction systems must be integrated to the interactive catalog platforms.

So, despite a high potential for both merchants and consumers to use AR for shopping, we must not get too far ahead of the cart with this one. What do you think? Is shopping with AR something you would do? When do you think it will be ready for prime time?

tags: ,
  • Rob Marchand

    This is interesting stuff – Layar is now availble on the iPhone, and that starts to make it more about the content (so instead of separate apps for different things, you select the overlays you want to see within a unified interface). Once you add context senstive recognition (context being the layers you have enabled), it could get quite interesting.

    There is also some nifty concept stuff shown here (full disclosure – I work for a subsidiary of Alcatel-Lucent).

    At first blush, I’m not sure to what extent I would use this kind of purchasing – but having said that, it brings the world of impulse buying to people remote from bricks and mortar – so maybe I would!


  • Great article — AR is mostly used now to recognize building, landmarks, and other things outdoors like you mention. It’s cool to see how it can be applied inside a building, where GPS signals aren’t as strong. Another potential application for indoors is use inside museums, where AR apps can take the place of headphone-led tours. The iTacitus project was launched with that goal in mind. I wrote an “Ask the PropellerHeads” column for that and tons of other uses. You can read the accompanying blog post here:

  • Hi Chris, thanks for your comment. Are you affiliated with the iTacitus project? I am aware of it, of course, since I’m based in Europe and these EU projects are very important in driving technology research.

    I have iTacitus listed as a developer of AR applications and content but I don’t think they will release anything commercial. Unfortunately, technology transfer from EU research is extremely low.

    In your blog you asked which company or group came up with the first working application of this technology on a phone. This is not a mystery at all, really. It just depends on what you call a mobile phone!! and what you consider to be Mobile AR. There were many examples using 2D markers before Mobilizy launched Wikitude (the first application designed for consumers which uses GPS and compass for the user localization and tracking).

    Please visit the Official ISMAR Mobile committee’s History of Mobile AR here: Developed by Daniel Wagner, one of the experts in the field and a Researcher at Technical University of Graz (Austria).


  • Very much enjoyed reading this article.

    Regarding the second scenario (2nd paragraph) in your article, with the shoes:
    There is actually a Startup company with a visual search application, where instead of searching by text (Google), you would search by picture.

    In this case (using a mobile) it would be a search by the photo you just took of the guy’s shoes. It will analyze all photos already located in their database taken from lots of sites and show you the same pair of shoes with online/shop prices, as well as very similar shoe items.

    The Startup is called Superfish (

  • Great article Christine.

    Augmented Reality will enhance offline life with any kind of digital (meta-)data that can enhance the information/experience on a certain action/subject.

    I would shop with AR, but I’m interested in understanding the privacy which are effected by AR.

  • Hi Gianluigi, You raise an excellent point which I hinted to (when saying that the user turns off the sound which cameras make when taking a photo before snapping a photo of the shoes)but have purposely avoided addressing in the post.

    I don’t treat privacy in this post for two reasons: it is a rich and complex topic all on its own, possibly for a future post (the problem of privacy is not solely for Shopping use scenarios) and, in the shopping scenario, the simplest solution is for the application to only capture and send the recognized “features” of the image, never the full image itself. This “feature extraction” process is vital to the future of image recognition.

    If processing (any feature extraction, for example) is done in the cloud, there could be a “privacy filter” either in the handset or in the cloud. Imagine something as follows: any photo or video which is used for identification/tagging purposes should first detect faces. Then obscure/blur any face in the image before storing it in a database for an application. We might go about this in a different way which is to request that the user highlight only that part of the image captured which they want to know more about. If it has facial features, it gets rejected.

    Having users select points of interest is another important topic. We would run the risk of the application taking much too long to retrieve relevant information because the software does not know precisely which object in the image is the highest interest to the user. This question of multi-object recognition (and detecting what the user is “really” looking for) is already a significant challenge with the location-based Mobile AR apps. It is something which, if readers were ubiquitous and integrated in handsets, RFID tags could help with in a shopping environment.

    Many thanks for your comment!


  • this is the first time ive heard of AR had to read the article to find out what it was good article
    as usual i trying to figure out how i can make a few pennies on it thanks

  • scmoatti

    Very interesting post.
    Nokia offers a similar service, based on real-time image recognition and barcode recognition, called Nokia Point & Find. Check it out at