Context Aware Programming

I’m increasingly realizing that many of my gripes about applications these days are triggered by their failure to understand my context in the way that they can and should. For example:

  • Unruly apps on my Android phone, like gmail or twitter, update messages in the background as soon as I wake up my phone, slowing the phone to a crawl and making me wait to get to the app I really want. This is particularly irritating when I’m trying to place a phone call or write a text, but it can get in the way in the most surprising places. (It’s not clear to me if this is the application writers’ fault, or due to a fundamental flaw in the Android OS.)
  • Accuweather for Android, otherwise a great app, lets you set up multiple locations, which seems like it would be very handy for frequent travelers, but it inexplicably defaults to whichever location you set up first, regardless of where you are. Not only does it ignore the location sensor in the phone, it doesn’t even bother to remember the last location I chose.
  • The WMATA app (Washington Metro transit app) I just downloaded lets you specify and save up to twelve bus stops for which it will report next bus arrival times. Why on earth doesn’t it just detect what bus stop you are actually standing at?
  • And it’s not just mobile apps. Tweetdeck for Mac lets you schedule tweets for later in the day or on future dates, yet it defaults to the date that you last used the feature rather than today’s date!  How frustrating is it to set the time of the tweet for the afternoon, only to be told “Cannot schedule a tweet for the past”, because you didn’t manually update the date to today!

In each of these cases, programmers seemingly have failed to understand that devices have senses, and that consulting those senses is the first step in making the application more intelligent. It’s as if a human, on awaking, blundered down to breakfast without opening his or her eyes!

By contrast, consider how some modern apps have used context awareness to create brilliant, transformative user experiences:

  • Uber, recognizing both where you are and where the nearest driver is, gives you an estimated time of pickup, connects the two of you, and lets you track progress towards that pickup.
  • Square Register notices anyone running Square Wallet entering the store, and pops up their name, face, and stored payment credentials on the register, creating a delightful in-store checkout experience.
  • Apps like FourSquare and Yelp are like an augmentation that adds GPS as a human sixth sense, letting you find restaurants and other attractions nearby. Google Maps does all that and more. (Even Google Maps sometimes loses the plot, though. For example, yesterday afternoon, I was on my way to Mount Vernon. Despite the fact that I was in Virginia, a search unadorned with the state name gave me as a first result Mount Vernon WA, rather than Mount Vernon VA. I’ve never understood how an app that can, and does, suggest the correct street name nearby before I’ve finished typing the building number can sometimes go so wrong.)
  • Google Now, while still a work in progress, does an ambitious job of intuiting things you might want to know about your environment. It understands your schedule, your location, the weather, the time, and things you have asked Google to remember on your behalf. It sometimes suggests things that you don’t care about, but I’d far rather than than an idiot application that requires me to use keystrokes or screen taps to tell the app things that my phone already knows.

Just as the switch from the command line to the GUI required new UI skills and sensibilities, mobile and sensor-based programming creates new opportunities to innovate, to surprise and delight the user, or, in failing to use the new capabilities, the opportunity to create frustration and anger.  The bar has been raised. Developers who fail to embrace context-aware programming will eventually be left behind.

tags: ,

Get the O’Reilly Programming Newsletter

Weekly insight from industry insiders. Plus exclusive content and offers.

  • Robert Lucente

    Perhaps true “artificial intelligence” eventually fails because software is terrible at determining context. BTW, there is nothing wrong w/ point solutions. You have to start somewhere.

    • leodbis

      actually it is quite accurate in lab now. I would say 78- 98 percent for our experience with several set of not perfect data. No GPS only cell Id, etc.
      It is quite ok for some applications like: automatically reply I am driving and will call you back in {X} minutes.

  • Steven Limmer

    Ux testing; often overlooked and undervalued. However so many apps are developed by folk with a particular need, for themselves, and in their own time; that touches like context awareness simply aren’t considered as a story in the development process.

    Have you provided feedback or reviews to the ‘offenders’; I’ve found that raising issues via the feedback process gets them resolved quite quickly.

  • http://btlr.com/ Jacob

    we demand more and more from the software, sometimes much more than was planned by developers. And sometimes developers plan A while users find B,C.. Z ways to use the app. This issue is partucularly important also for me as a developer. I need to see how users actally use my app. I will try adjust the app accordingly, but to do that i need users’ inputs.

  • Murray McKercher

    It used to be said that content is King. I believe Radar’s article indicates that perhaps now, “content in context is King”…

  • Greg Howe

    As a solution developer I understand both sides. But keep in mind many of the points in the article show a PERSONAL experience; grab 100 other users of those apps and you know they won’t all agree. Expectations are now becoming “meet MY needs”. Devs can do better at the start but this is a new way of looking at building software and change takes time. Getting it right in v1.0 is difficult. Good companies will adjust to this new way of rapid development with constant adjusting based on user feedback.

  • Srinivasan Ramaswamy

    Its all about My Data, My Solution, My Experience and My Context

    We at Ramco offer Rich User Experiences through Industry Leading ERP/HCM solutions. Our Solutions |are Context Aware (which also includes location). While Context includes geo location, we appreciate it even more. Just like my app being aware of where I am, it also needs to be aware of my usage pattern, my typical day, my work behavior, my usage context, my organization hierarchy and above all respect my privacy as well. Examples ??

    Some Samples of our Geo Aware Solutions include:
    Location Based Timesheet Checkin / Checkout,
    Location Based Workforce Management
    Location Based Sales Dashboard
    Location Based Customer Experience Management

    Being context aware, our Solutions show parts lookup based on frequency of usage, leave types based on balance availability, spares suggestion based on alternates available, manpower replenishment based on resource availability in the geographic proximity, .. the list goes on..

    Visit http://www.ramco.com to experience MUSIC (Mobility, User interface, Speed, In memory, Context aware).

  • http://acroyear70.livejournal.com/ Joe

    ah, but when is geo-aware software aware of too much? There was an article recently about the privacy issues around geo-tagging photos. Take a photo (with your GPS on) of your kid in your daycare, throw it up on twitter, and instantly EVERYBODY can look up what daycare it is from the geotags in the EXIF. Is that something you really wanted to make public?

    So too, any of these others. Every app that makes it public where you are also makes it very public when you are not at home? Someone can follow ones twits for a bit, figure out the pattern, find the home from a “taken at home” photo’s GPS tag, and instantly know when to hit the house.

    Too much information, perhaps?

    • Srinivasan Ramaswamy

      Exactly!! The line between privacy and capability has been overstepped very frequently. I would not want to push too much of data of where I am. As a user, I should have the option to decide on when and what to publish.

      Striking a balance is the key..

    • http://btlr.com/ Jacob

      that’s why we thought of co-location based service rather than location. We group users who are in the same location wighout knowing where on Earth this location is. And yes, we strip EXIF data from users’ pictures :) We are krowdthink.com

  • floatingbones

    “I just downloaded lets you specify and save up to twelve bus stops for which it will report next bus arrival times. Why on earth doesn’t it just detect what bus stop you are actually standing at?”

    When I lived in the Boston area, I could take 3 different subway lines (Red, Orange, Green) to 4 different stops to catch busses home. I don’t care about the closest subway stop; I care about which walk/subway/bus/walk trip will get me home the soonest. Perhaps the Google Now services would know how to compute the connections with high reliability — or present me with the pertinent information to let me make my own decision on which subway line to board.

    I am concerned with how much information Google Now could extract/market from my travel plans…

  • Bob the TOOLSmith

    I think the problem with the background apps is that it should be IDLE, not ACTIVE, background. Using idle time to update is fine, but competing with the foreground app for CPU is not!

    As for Geo-Aware… don’t exceed what you actually know. The Geo-Aware features that use IP Geo-location are so maddening! I have 3 devices lacking GPS in which apps, and especially browsers, are constantly showing totally wrong location info due to the presumed location. The errors are weird, too. If I connect from home, I get the Geo-location of the cable company’s gateway… 50 miles away. If I connect via WiFi hotspot, I get the Geo-location of the WiFi’s gateway… 30 miles away. And if I connect from work, I get the global network gateway in Virginia… 850 miles away. Can you say TOTALLY USELESS? And many apps/websites won’t let you set a different location, which means I just stop using it. Programmers: If you don’t see a GPS in the unit, DO NOT assume that the location can be known.

    Both of the above point out the most galling deficiencies today: failure to get enough user input, and failure to TEST. Your customers should not be your Alpha testers!

  • http://www.geo-tel.com/ GeoTel

    I feel you just voiced every frustration I have throughout the day with my Android. I wish I had much more control over the Idle/Active setting of background apps, especially when I travel abroad and am getting killed with international data charges. I still need these apps, but I want to control them more.

    And the whole GPS awareness feature is absolutely maddening, like Bob said. You really feel like they should be able to do a better job at seeing where you are and doing a little thinking (closest/fastest bus stop).

    I have been VERY surprised and pleased by Google Now. It still has some hiccups, but when it works, it is fantastic. I love that it’ll send me an alarm telling me what time I need to leave to catch my flight (also verifying that it is still on time) based on live traffic updates. And it does all this without me having to upload my flight information somewhere, it simply reads my emails and picks out the vital details.

  • http://www.softwarecandy.com softwarecandy

    It’s about time. It takes time to deliver new & improved features. Most app developers hardly make a penny on their apps and considering that app development must be done in their “free time”, this can take even longer. Providing built-in such features in the OS’s API may accelerate delivery time but it will still take time.

    • Charles Ferguson

      No, its about good design. These aren’t features that have just now become available. The functionality isn’t missing because it didn’t exist when the apps in question were developed, it’s missing because of a failure of imagination by the developers (and in some cases – where the function has been implemented reasonably widely elsewhere – because of outright poor design).

      • thanatoid

        No Charles, it IS often about time. Your suggestion that it is only a matter of design is willfully ignorant. It takes more time to develop an app that checks the GPS location and filters or chooses data by location than it does to not, even though the API has the ability to get that data. It doesn’t just magically appear. You may not agree with the decisions of programmers who choose not to take the time to utilize these features, but that doesn’t change the fact that they all require additional development, and therefor time, to leverage.

        • Bunswalla

          That’s a bit like saying – I would’ve designed this new car to be easier to drive, but I didn’t have time so I didn’t worry about the suspension. People will still get where they want to go, but the ride will be bumpy.

          • Catahn

            I’m going to have to go with the good design and fundamental understanding of human interaction. I am working on a major project for a large corporation and I have to tell you the UI is a pos. Aside from being buggy it is largely non-intuitive.

            When I as an automated tester bring up some of these non-intuitive points that will lead to frustration, I am frequently told “We will wait for customer feedback on that point.”

            At this point time has become the enemy because the up front design was outsourced and very little attention was paid to how a customer might actually want to use it.

            It may just be me because I have an innate sense for the flow of things. That said it bothers me how many little things could be improved with very little time and increase the overall satisfaction of the user experience.

          • Matthew

            Your concerns are completely valid, however aren’t these decisions made by managers and not developers? That’s how it works were I work as a developer.

          • metz2000

            @Bunswalla:
            Cars have millions of compromises on quality and feature because people couldn’t pay all those features.

            Is multi-zone AC better than manual or no AC? Yes it is but how many cars include it by default? And in Europe AC is not even a standard accessory, in many cases it’s an extra.

            And your example the suspension. A good suspension is self adjusting, but how many cars have suspension like that? Only a few. Some cars are still stuck with 20 year old designs.

            I could carry on listing hundreds of issues.

        • Charles Ferguson

          That’s a reasonable point, although its somewhat at cross purposes with my reading of the post I was responding to: “It takes time to deliver new & improved features.” My reaction to that is: these features aren’t new, and they should be considered the default baseline unless you fail to grasp their significance when you first build the app. (Note I said “default baseline”, not “must have at all costs”.)

          Leaving out context awareness because you’ve looked at it closely and made the hard decision to leave it out is 100% valid – the market will tell you if you made the right choice.

          Leaving it out because you don’t grasp that it is (or could be) a game-changer is not something a designer should be proud of.

  • david

    It’s not just Android. Trying to text my bf Clint on an iPhone and typing “Cli” in the destination, the #1 result is his work phone (which of course does not receive text messages and I never used for texting), then a bunch of results of people I rarely text, then, at #5, the correct number – which I text daily. It shouldn’t be difficult to sort out numbers that are not mobile (“Work”) and sort other numbers by popularity. I get the feeling that Apple’s QA has been deteriorating, and it’s interesting to hear similar stories from the Android world.

  • Kerosene

    Let me bring up a new point: A lot of these context-sensitive and performance issues wouldn’t be nearly as much of an issue (at least not for as long a time) if the designers and developers themselves were active users of their applications. Too often, the developers are just implementing what they’ve been told to do, with little opportunity to extend the design before the targeted ship-date. Achieving basic functionality tends to take precedence over raw performance and user experience (and security!). That starts to change if developers truly understand the capabilities of their platform (the platform API documentation is their “bible”), use the application in their real lives that simulates the lives of their target users (e.g., they’re on the road also), and have an allocation of time to implement and show what’s possible without having to justify every enhancement.

  • Annoyed

    So you bought or got a product free then want to complain it doesn’t fit your exact needs/wants…. either pay somebody to write you what _you_ want or accept that in the triangle of feature, quality, and time you can only fix two corners, time is usual fixed so that means low quality with loads of features, or a few well designed ones, or somewhere in the middle.

    Its like Open Source, either take what’s there and feedback or write your own… Don’t curse the poor dev who has done their best to deliver something usable to you for not spending another xx mandays to add in some feature that’s of marginal value

    • Matthew

      As a developer, thank you!

  • E.T.

    I have found some fake-GPS-location programs handy – not to hide my location but make my phone faster. I haven’t found a perfect one yet, they tend to ask too much permissions and are potentially dangerous.

  • 98188

    Systems of Engagement is the term used to describe what you are asking for. Google Now is a great example of a System of Engagement. It is different than a System of Record most developers are used to developing. It involves maintaining a context of data, reacting to changes in the context, brokering data from Systems or Record for a data mashup in the context, and pushing commands and data to the UI. Your device’s sensor data is one part of your context.

    The problem is that most developers are just not aware of what a System of Engagement is and how one is developed, especially at scale.

    Anyhow, I’ve worked in this space for a while and Forrester Research describes it well in “The Future of Mobile Application Development” by Jeffrey
    Hammond and Julie Ask.

  • mlportersr

    I think there are two issues in play here. First, I think there is an issue with the way programmers are trained. Dr Lynn Stine wrote about this problem quite clearly, and prophetically, in a paper she wrote in 1999(!) about “Challenging the Computational Metaphor”. We aren’t teaching programmers to think about applications in a way that even allows them to recognize that there is a problem.

    A second, possibly related, issue is that there are too many people writing code that should be flipping burgers at McDonald’s. I’m talking about people who don’t really care about the work on any level beyond it being a way to obtain a paycheck. And matters get even worse when one of these folks contrive to get themself into a “supervisory” position.

    PS: Why do I suddenly find myself having to resist the urge to use phrases like “young whipper-snapper”? Hmmm….

  • mike

    The bulk of the complaints you have seem to me to stem from inadequate testing. It’s all very well to test an app in isolation, and have it pass all those test cases, but if you have no test cases that cover interactions with other apps, or don’t cover date or location changes, it speaks of inadequate test coverage.

    • Matthew

      Are you willing to pay $$$ per app so that we developers can afford to fly around the world to test all of these date/time/location/context situations for you?

      Yeah, I didn’t think so.

      At my company, we do the best testing (and it is pretty darn good) we can with the resources we have.

  • Paulo

    Applications using GPS positioning consume more battery – most people don’t have enough juice as it is and turn off their app’s ability to utilise location based services. Why build it in, when you’re building on a shoestring, so that users turn it off !
    Someday we’ll have good batteries and this will be more relevant.

    • Andrew

      GPS doesn’t really use a lot of battery, your screen does, but typically if you are using an app with GPS, you are keeping the screen on the whole time. Guilt by association.

      • Matthew

        That’s not entirely correct. One GPS location check may consume little power, but the author and several other commenters are suggesting that they want very frequent location checks. As a developer, I tested an ARG app that did frequent location updates. I put an iPhone 4s with a fully charged battery, running only the normal system processes and this app, in my pocket with the screen turned off. When we set it to update the location every 30 seconds, the battery was dead in 4 hours. When we changed the interval to every 60 seconds, the battery was dead in 7 hours. That is without ever using the phone for anything else. The average consumer’s phone probably would be dead in under 2 hours with location updates every 30 seconds.

        • Brian Tarbox

          Your point actually confirms the original post…most of the time our location is not very dynamic and so infrequent GPS checks are appropriate. However, the system could/should notice when I’m in motion and appropriately calibrate GPS update frequency. Its exactly the sort of thing a “location aware” device should do.

          • Matthew

            How do you propose that the system notice that you are in motion? I am only aware of three technologies currently in many smartphones that could potentially “notice” when you start moving. However, all of them have flaws in how they might be used to detect movement in the manner you describe.

            The first is GPS. Obviously using GPS to “notice” when you start moving and to then turn on GPS isn’t going to work (Chicken or egg argument?).

            The second is by using cell service and Wi-Fi signals, where based upon recognizing different networks or triangulating distance from cell towers, motion could be inferred. However, this has many problems as well, such as the fact that due to electromagnetic interference, insulating effects from building materials, the number of other users in proximity to or using the same connection, and other factors, the strength and reach of these signals varies all the time. Meaning that you could be sitting at your desk and your phone might intermittently pick up and lose or read increases and decreases in various signals and then incorrectly interpret that as movement and then potentially wake up GPS and start consuming battery life.

            The third is the accelerometer. This technology has the best potential for what you want but also has potential issues. Let me give two realistic examples. Some people like to wear their phone while they exercise on a treadmill (listening to music). Some people fidget, i.e. bouncing their leg while sitting in a chair (with their phone in a pocket). In both of these cases they aren’t going anywhere, but the accelerometer will register movement and then potentially wake up GPS and start consuming battery life.

            My point of all this is that there simply isn’t a technology currently built into most phones, that I am aware of, that would achieve what you want in the way that you want it, without potentially still draining your battery.

          • Brian Tarbox

            I’m suggesting that a device could check GPS every few minutes (which your post suggests would consume little power) and use that to notice movement. I typically sit at a desk for hours at a time..and my commute home takes about an hour so it seems a device should be able to notice that transition with a low cost / low frequency GPS check.

            In the case I was actually wondering about I had my phone’s mapper application open and running…and IOS location services still didn’t notice our arrival at the train station..even when the map showed us there. Not sure what location-aware is even supposed to mean in this case.

          • forbin

            Well, there’s going to be SOME usage of GPS; you simply cannot get away from it. But there are adaptive algorithms that can be used to be smart about how often you wake up and when. For example, when waking infrequently, don’t just wake, take a snapshot, and conclude whether the device is moving or not. Wake, take 2 or 3 samples across a moderate period, and estimate a travel rate and a “direction cone.” Thereafter, wake at appropriate intervals to confirm travel or to realize a new motion (or “motionless”) estimate is needed. This also extends to adaptively characterizing the accelerometer readings. Storing a history of these would let your program learn the difference between the pattern of a bouncing leg (and no signification geoposition change) and a continually accelerating/decelerating commuter train ride. (Hmm, also noticed your past 4 locations are all on a train track. Could that be a clue?)

            The problem is that even among smart people, there are precious few developers capable of designing and programming this way ALL the time. And few managers willing to allow a little extra time for it, because they don’t understand how important it is for the app to be smart about how it manages itself. “Oh, we can always fix it later.” Even though other parts of the design may come to rely on that specific (and broken) “first draft” behavior.

          • forbin

            [Typos. c/signification/significant/. ]

          • leodbis

            Matt,

            There are motion detector built in your phone sensors. Google did not reveal the API in android system. The battery consumption of the motion detector is almost zero.

  • http://www.xnonymous.com/ David Peterson

    Which is exactly why some programmers have sets of projects spanning many years until they release even one. Especially if they are young, in college, have a girlfriend, working on them on their own, and trying to stay sane….Even when you work in groups who share the same characteristics as mentioned above, collaboration times are not consistent due to personal schedules, and leads to delays. You’re last sentence hits the nail right on the head, the user must feel a sense of immersion when using your software, in that there is liquid flow in transitions and available features…..Like Bruce Lee said, “Be water my friend”

  • robert hedlund

    the city maps often do not have a straight forward relation to gps cords. call your councilman and complain to him/her.

  • http://www.davidsoninfocussystems.com/ Beresford Davidson

    My Android doesn’t even load up anymore. The factory settings don’t even reset. All I can see is pictures in still and no water reflections. I tried to get in touch with the company but they are too busy to respond. They gave me address of the manufacturer in China. He has yet to send me a mail from China telling me what to do or where to go to get a new SDI chip. The Android promises a lot of good works but they need programmers to do the do. Guess what, because I am an old time programmer and tech-writer out of work, they won’t even hire me to fix their faults. Microsoft Surface going kick Google butt.

  • Brian Tarbox

    This is exactly right and no more so than in the very apps you’d least expect it. If I’m using Waze or Google Maps to navigate and I ask for “Gas” how about showing me gas stations along my route? Does me no good to show a “nearby” gas station a mile behind me on the highway! And if I’m in Boston and ask for Main St. assume I mean Main Street in Boston not in Tucson or some random city. How are such glaring failures getting through design?

    • Daniel

      waze and gas … been there done that while running on fumes … aggravating

  • Paul Muench

    As a counterpoint, sometimes I wish I had a context free button as I’m exploring the digital world for context “free” information.

  • Michael Barnathan

    I not only agree, but consider this the reason why services such as Foursquare are failing. Explicit check-ins? Come on. Poll the GPS and check in automatically.

    I’ve bet on this strategy myself: I created an app called Clipless (http://clipless.co) a couple of weeks ago which can sit in the background, periodically check the location, and alert the user only when there’s a deal in the user’s specific location, within a radius that the user sets. It’s too soon after the launch to tell whether it’s catching on yet, but regardless, I intend to design all of my future apps with this type of contextual interaction in mind. It makes for a much more streamlined user experience.

    • Brian Tarbox

      The radius thing is key. I almost love the iPhone location based service except that I can’t set the radius. I’ve tried to set an alarm on the location of my local commuter rail station so that I don’t sleep through it. The problem is that the location alarm never goes off and I can’t tell why. Are we not close enough…long enough for it to notice? There seems to be no way to tell.

      • strattonp

        if you’re in a train – do you have GPS signal. Just because GPS is on doesn’t meet it knows where it is.

  • RonK

    Mostly due to poor programmers that code directly from the requirements document without trying to understand the business and what it is they are creating. They never think outside of that document from the perspective of someone that will actually be using what they are creating and therefore never take that extra step to be creative and improve on the product. I see it all the time, but hey, they keep me employed :)

    • fowl_bob

      I don’t blame the programmers. Instead, would prompt management to put more effort into identifying the requirements. The process model may also be an influence. The waterfall method requires that all requirements be identified sufficiently to code the last word in product excellence whereas other models define frequent releases that implement product improvements which cannot be included due to extreemely short release cycles. The rate of system changes seems to be pushing the move to abandon waterfall because it cannot keep up with the change in operating constraints as well as methods that adapt to rapidly changing requirements. The shortcomings in applications mentioned in the article are based on mobile communication device use. This is an area of rapid advances in system feature support. Fine tuning requirements before initial release is a poor choice compared with short development and release cycles that typically get partially complete products to the market quickly. Which do you want, a rough product now or the best in class two years from now? Market share grows quickly from early adopters giving those products a lead over late arriving compititors.

  • Terrence

    DO blame the programmers! I develop software, and as long as business insists upon hiring third-worlders to do the work, they’re going to get third-world results. And there’s simply nothing for that in this environment.

    • RonK

      thanks for pointing out the part I cautiously left out ;)

  • Tom McQueeney

    Regarding your statement that “programmers seemingly have failed to understand that devices have senses, and that consulting those senses is the first step in making the application more intelligent,” I agree with several of the comments that jump to the defense of the programmers. E.g. “Kerosene” points out that most programmers are given feature sets — and more importantly, deadlines — defined by others. Most developers I work with are eager to build quality applications. But quality is only one winning factor in the marketplace. Product owners want shipping software at a reasonable cost.

    I suggest your conclusion that “Developers who fail to embrace context-aware programming will eventually be left behind” is not as accurate as saying intelligent, context-aware applications will be the norm only when competition, or users with their wallets, demands it.

  • strattonp

    Knowing where you are is nice – but sometimes – like checking a bus schedule to see how to arrive at the bus/train stop at the same time – forward planning.

  • http://specialbrands.net/ Daniël W. Crompton (webhat)

    If Drew Crawford is to be believed the fundamental flaw in the Android OS is the Garbage Collect.