You are sitting in a café reading the newspaper and there’s a QR code on the page near an article which you found interesting. You’ve never seen this box symbol in a newspaper before but, since you already know the general purpose of these funny black and white checkered boxes, you ask yourself what this is about. You start up your QR code reader application (yes, it’s pre-installed on many camera phones already), aim, see the digital box appear on your screen around the QR code and snap a picture. Off goes the code and, depending on your mobile network connection, somewhere between 30 seconds and 2 minutes later more details about this feature story appear in your mobile browser screen.
Time passes (months, maybe years) without your noticing. One day, you’re sitting in a café reading the news on your phone and you notice that the person next to you has some really nice footwear. No QR code on the neighbor’s shoes? No problem. You start up your visual search application (no, it’s not available on all handsets), act like you are trying to find something on the newsreader screen while you turn off the camera’s “snap the photo” sound, then discreetly aim and take the photo of the shoes. No one has noticed, right? You wait, you act like you’re still reading the news. Your mobile browser opens and on your screen is the exact model of shoes on your neighbor’s feet. Another click and you can check the price and availability from stores nearby.
The world is an interactive catalog
The first Sears catalog was published in 1888 for farmers who came to town only rarely and needed to place orders with someone other than their local merchant. By 1894, the Sears catalog had grown to 322 pages, featuring sewing machines, bicycles, and even cars. The next year it exceeded 500 pages and contained more products than people had previously thought existed. About 100 years later, the World Wide Web was getting off the ground, and in the past 15 years we’ve witnessed the explosion of information and had our eyes opened to a vast universe of merchandise, including customized versions of many items made on demand for our pleasure.
Augmented Reality shopping
Much like the farm wife with her Sears catalog, consumers will be able to use simple AR applications to make more informed buying decisions.
Items that are fixed in place (a building, for example) are well suited to research and information display using GPS and compass-based mobile AR as we know it today. Shopping for a new house or apartment to buy or rent? Among the original 4 or 5 layers in the mobile AR application Layar, is a real estate application. Via the Layar partner Funda, Layar launched with a few dozen properties. Now, with 17 real estate partners, tens of thousands of properties are listed worldwide. Perhaps, with just a few more lines of code, could the entire US Multiple Listing Service database be accessible from Layar or a similar application?
AR shopping for objects which move or in indoor environments (where GPS doesn’t work as well, if at all) is a little more challenging. But merchants and brands are always looking for novel ways to enhance a retail experience, and AR technology is on the list of tools that could serve the customer. In 2006, Michigan State University researchers supported by a Microsoft Research grant designed the PromoPad system, a shopping assistant based on a TabletPC which would ride in the shopping cart “cradle.” Since image recognition was not sufficiently advanced to serve as a way to identify objects (and this was not the focus of their study), a combination of fiducials (another name for 2D bar codes) and RFID were used to detect the objects around the PromoPad user. The team put together a user interface for simple browsing, thought about how to attractively overlay information on the real world using a video camera connected to the TabletPC and how to design the shopping environment to make sure that only the items most likely to be of interest were presented to the shopper.
Using similar concepts, Insqribe is offering what appears to be a commercial version the PromoPad. They call it a real-time proximity-based marketing system. One version of Insqribe’s system still requires a marker or 2D barcode for recognition. They have also implemented a version with AR features. Another example of 2D barcode-based AR shopping is provided by Moving Brands. The current version of Moving Brands does not use the natural features of an object to recognize the product which has the attention of the shopper.
Image recognition as the basis for obtaining information in an AR-enhanced retail application is here today, but only for a few classes of products. Books, CDs and DVDs are the low hanging fruit for this application because the image databases for these searches are already on-line and the “planar” objects (flat) are far easier to recognize than 3D objects. One of the iPhone applications for the books, CDs and DVD search application is provided free by Swiss start-up kooaba. In June 2009, Amazon’s A9.com division acquired another company leading in this domain, SnapTell. Although I haven’t tried it personally, I’m told that the technology is already in Amazon’s service. [Note: please don’t confuse the technology which this post is about with Microsoft’s Bing Visual Search. Same term, Bing is a close “cousin” of image search for retail.]
In the case of books, CDs and DVDs, you might be asking yourself why you would need this application if the real object (e.g., the book about which you want more information) is already in your hand. Some of the answer lies in formats. Maybe you would like to purchase and read the digital version of the book you’re holding. Maybe you would like to listen to the music on your iPod but you are holding the CD. Perhaps you would like to see the movie you are holding in DVD format on a large screen in a theater in your neighborhood.
Good news or bad news?
Like the Sears catalog over a century ago, AR with visual search will cause the minds of people around world to open to entirely new possibilities. Sounds promising? For many people, having the world around them in the form of a searchable, interactive catalog is a distasteful, repugnant reminder that everything is for sale in a hyper-materialistic society. For others, it will become an addiction, perhaps causing people to buy beyond their means.
There is also the bright side of turning the world into an interactive catalog. Motivations for using AR for shopping could include the need or passion to be a frugal shopper. The applications will be able to support price comparisons and, using GPS, perhaps recommend a shop for immediate purchase. Too busy to go into a store to browse? Some will embrace mobile catalog shopping for the same reason they prefer on-line shopping: busy lifestyle. There are also those consumers who have a keen desire to adopt more careful, let’s say “informed,” habits with respect to their consumption of physical goods and services. They want to purchase and consume materials which they know to be wholesome, damage the planet as little as possible, and may even achieve some social good. An interactive catalog based on images could alleviate the need to get a bar code or to have the exact model number of a product.
And, adding this new functionality will permit companies to explore new business models. Would consumers be willing to pay a premium for a faster or higher-accuracy visual search engine for retail? Would consumers be willing to pay a premium to have reminders sent to them when they are near a retail outlet where a previously “spotted” object is available in their price range?
Try it on for size and style
AR is also useful when combined with a PC or another device in a virtual dressing room application. For example, Zugara is proposing to develop for its clients customized versions of a Webcam Social Shopper application which allows prospective customers to virtually try on clothes and use motion (actually gesture) detection algorithms for keyboard- and mouse-free navigation (another use for computer vision). Cisco has just released a short commercial vision video which shows how such technology will help shoppers in the future.
This virtual catalog combined with a dressing room concept is attractive but there remain some questions. What happens if the person using the application is a different size than the model on which the garment was originally cast? Are there different markers for people with different height and weight? Then the key question is if this indeed increases retail sales as Zugara suggests.
Feet back on the ground
For many of the location-based shopping services available today, such as recommendations of restaurants and places to hang out, we already have examples of the use of AR (see for example, the Monocle feature of the Yelp iPhone application and, in Japan, Tonchidot‘s Sekai Camera). We should see AR integration as a feature of more navigation and recommendation services in the future.
Although image recognition technologies for AR applications on mobile handsets are not quite ready to take on the challenge of identifying the pair of shoes your neighbor is wearing, they are entirely up to the task of identifying, from a well-lit photograph, the model of a car or a logo. Expect retail outlets and brands that provide fast-moving consumer goods to be among those eager to exploit mobile AR for shopping.
Shopping with AR is a shoe-in for some, but not all things we might want or need. There remain gaps in this ecosystem which will take a few years to fill. For example, the consumer may wish to consummate a purchase directly from the application which has an AR feature. Mobile commerce is far from a fait acompli. Transaction systems must be integrated to the interactive catalog platforms.
So, despite a high potential for both merchants and consumers to use AR for shopping, we must not get too far ahead of the cart with this one. What do you think? Is shopping with AR something you would do? When do you think it will be ready for prime time?