The home automation paradox

Humans are messy, illogical beasts — we must create systems that expect us to be human, not punish us for when we are.

Editor’s note: This post originally appeared on Scott Jenson’s blog, Exploring the World Beyond Mobile. This lightly edited version is republished here with permission.

The level of hype around the “Internet of Things” (IoT) is getting a bit out of control. It may be the technology that crashes into Gartner’s trough of disillusionment faster than any other. But that doesn’t mean we can’t figure things out. Quite the contrary — as the trade press collectively loses its mind over the IoT, I’m spurred on further to try and figure this out. In my mind, the biggest barrier we have to making the IoT work comes from us. We are being naive as our overly simplistic understanding of how we control the IoT is likely going to fail and generate a huge consumer backlash.

But let’s backup just a bit. The Internet of Things is a vast sprawling concept. Most people refer to just the consumer side of things: smart devices for your home and office. This is more precisely referred to as “home automation,” but to most folks, that sounds just a bit boring. Nevertheless, when some writer trots out that tired old chestnut: “My alarm clock turns on my coffee machine!”, that is home automation.

But of course, it’s much more than just coffee machines. Door locks are turning on music, moisture sensors are turning on yard sprinklers, and motion sensors are turning on lights. The entire house will flower into responsive activities, making our lives easier, more secure and even more fun.

The luddites will always crawl out and claim this is creepy or scary in the same way that answering machines were dehumanizing. Which, of course, is entirely missing the point. Just because something is more mechanical doesn’t mean it won’t have an enormous and yet human impact. My answering machine, as robotic as it may be, allows me to get critical messages from my son’s school. That is definitely not creepy.

However, I am deeply concerned these home automation scenarios are too simplistic. As a UX designer, I know how quixotic and down right goofy humans can be. The simple rule-based “if this, then that” style scenarios trotted out are doomed to fail. Well, maybe fail is too strong of a word. They won’t fail as in a “face plant into burning lava” fail. In fact, I’ll admit that they might even work 90% of the time. To many people that may seem fine, but just try using a voice recognition system with a 10% failure rate. It’s the small mistakes that will drive you crazy.

I’m reminded of one of the key learnings of the artificial intelligence (AI) community. It was called Moravec’s Paradox:

It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.

Moravec’s paradox created two type of AI problems: HardEasy and EasyHard.

HardEasy problems were assumed to be very hard to accomplish, such as playing chess. The assumption was that you’d have to replicate human cunning and experience in order to play chess well. It turns out this was completely wrong, as a simple brute force approach was able to do quite well. This was a hard problem that turned out to be (relatively) easy.

The EasyHard problem is exactly the opposite: a problem that everyone expects to be simple but turns out to quite hard indeed. The classic example here is language translation. The engineers at the time expected the hardest problem was just finding a big enough dictionary. All you had to do was look up the words just plop them down in perfect order. Obviously, that problem is something we’re still working on today. An EasyHard problem is one that seems simple but never….quite….works…..the….way….you…..want.

I claim that home automation is an EasyHard problem. The engineer in all of us assumes it is going to be simple: walk in a room, turn on the lights. What’s the big deal? Now, I’ll admit, this rule does indeed work most of the time, but here are series of exceptions that break down:

Problem: I walk into the room and my wife is sleeping; turning on the lights wakes her up.
Solution: More sensors — detect someone on the bed.

Problem: I walk into the room and my dog is sleeping on the bed; my room lights don’t turn on.
Solution: Better sensors — detect human vs pets.

Problem: I walk into the room, my wife is watching TV on the bed. She wants me to hand her a book, but as the the room is dark, I can’t see it.
Solution: Read my mind.

Don’t misunderstand my intentions here. I’m not luddite! I do strongly feel that we are going to eventually get to home automation. My point is that as an EasyHard problem, we don’t treat home automation with the respect it deserves. Just because we can automate our homes doesn’t mean we’ll automate them correctly. The real work with home automation isn’t with the IoT connectivity; it’s the control system that will make it do the right thing at the right time.

Let’s take a look at my three scenarios above and discuss how they will impact our eventual solutions to home automation.

More Sensors

Almost every scenario today is built on a very fault-intolerant structure. A single sensor controls the lights. A single door knob alerts the house I’m coming in. This has the obvious  error condition that if that sensor fails, the entire action breaks down. But the second more likely case is that it just infers the wrong conclusion. A single motion sensor in my room assumes that I am the only thing that matters, my sleeping wife is a comfort casualty. I can guarantee you that as smart homes roll out, saying “sorry dear, that shouldn’t have happened” is going to wear very thin.

The solution, of course, is to have more sensors that can reason and know how many people are in a room. This isn’t exactly that hard, but it will take a lot more work as you need to build up a model of the house, populate it with proxies — creating, in effect, a simulation of your home. This will surely come, but it will just take a little time for it to become robust and tolerant of our oh-so-human capability to act in unexpected ways.

Better Sensors

This, too, should be soon in coming. There are already sensors that can tell the difference from humans and pets; they just aren’t widely used yet. This will feed into the software simulation of my house, knowing where people, pets and things are throughout the space. This is starting to sound a bit like an AI system, modeling my life and making decisions based on what it thinks is needed at the time. Again, not exactly impossible, but tricky stuff that will, over time, get better and better.

Read my mind

But at some point we reach a limit. When do you turn on the lights so I can find the book and when do I just muddle through because I don’t want to bother my wife? This is where the software has to have the “humility” to stop and just ask. I discussed this a bit in my UX grid of IoT post: background swarms of smart devices will do as much of the “easy stuff” as they can but will eventually need me to signal intent so they can cascade a complex set of actions that fulfill my goal.

Take the book example again. I walk into the room; the AI detects my wife on the bed. It could even detect the TV is on but still know she is not sleeping. But because it’s not clearly reasonable to turn on the lights to full brightness, it just turns on the low baseboard lighting so I can navigate. So far so good — the automatic system is being helpful but conservative. When I walk up to my wife and she asks for the book, I just have to say “lights” and the system turns on the lights, which could be a complex set of commands turning on five different lights at different intensities.

At some point, a button, an utterance, a gesture will be needed because human interaction is too subtle to be fully encapsulated by an AI. Humans can’t always do it; what makes us think computers can?


My point here is to emphatically support the idea of home automation. However, the UX designer in me is far too painfully aware that humans are messy, illogical beasts, and simplistic if/then rules are going to create a backlash against this technology. It isn’t until we take the coordinated control of these IoT devices seriously that we’ll start building more nuanced and error-tolerant systems. They will certainly be simplistic at first and still fail, but at least we’ll be on the right path. We must create systems that expect us to be human, not punish us for when we are.

tags: ,

Get the O’Reilly Hardware Newsletter

Get weekly insight and knowledge on how to design, prototype, manufacture, and market great connected devices.

  • Alex Tolley

    The backlash might happen quite quickly. We have a Nest thermostat and it can be infuriating to live with. I can just imagine what a home with many of these black box like devices will be like. And that is before they are maliciously hacked for fun and profit.

    • Doug K

      “We have a Nest thermostat and it can be infuriating to live with.”
      I looked into getting the Nest a year ago, but concluded it would be infuriating to live with..

      A programmable dumb thermostat has 90% of the functionality of Nest. The other 10% won’t work right (easyHard problem) 90% of the time. The software will need to ask, or to be told, to get it working. At that point I’d rather push a button on my dumb thermostat than engage in a long argument with the software to persuade it.

      In any case home automation is not working on useful functions. Turning on the lights isn’t hard, but I could really use some help with cleaning the toilets and stacking the dishwasher..

      • allan

        The main oversight in this article is the huge dark side of the wireless technologies and sensors and that is infinitely growing exposure to radiation to which many people are sensitive but unaware. Physicians are not trained adequately to recognise the EHS or Microwave Syndrome. People develop digestion problems and they are told it is gluten but in effect it is microwave sensitivity. People develop thyroid problems – in fact the world has thyroid pandemic at the moment but the health stats are hidden to allow profit-oriented cohorts who push new technologies seeing them as the best golden goose ever.Humans are not bags of minestrone sooup but they are treated this way by those who push such technologies as “wanted”. Many of us do not care about robots and sensors. And yet, we are being forced to accept them – they are imposed on us without anyone consulting communities. Cell and microwave towers outnumber trees on our streets!!! In fact, microwaves are quietly killing trees. That is what Scott needs to write about or try to grasp.

  • Guest

    Works great until I tell my wife. ‘Lights, on the car need to be fixed”. Star Trek forced the person speaking to start with “Computer”. This might be a good model when others are in the room.
    In many situations this was laughable as a person would be alone in a room. The computer would know this as everyone has a tracker on their person. Stating the words into a console “tea, earl grey, hot” by the sole participant of the room would be a good hint that they wanted the computer to make them tea.

    • Scott Jenson

      My ‘lights’ example was meant as an illustration only. There are many many ways for the voice action to be less ambiguous. The broader point was having some means to show intent.

  • Boltar

    Your example hints at a bigger question: are there not cases where our image of “automation” is driven more by immature sci-fi fantasy rather than practical and truly useful functions? Maybe turning lights on in a bedroom is one of those tasks that is better addressed through a simple (or even a somewhat intelligent) switch. Is manually activating a light in a bedroom really a problem for most people? I’m not saying even that all automated lights are useless, but I do think it’s important to that automation provide a compelling advantage (without constant inadvertent consequences) in order to be more than just an entertaining novelty.
    It’s also important to remember that humans are animals whose requirements for good health simply don’t evolve quickly. The innovation of driving everywhere is an example of a modern convenience that has damaged public health because we failed to grasp the importance of physical exertion.
    I’m also not a Luddite, but I think as important as the technical advances that make automation possible is exerting thought and planning so that automation actually improves quality of life.

    • Scott Jenson

      No, it’s an interesting question to ask. Every technology offers new choices and potentials. If we think of it in ‘the old way’ it doesn’t really make sense. So to your case, if we think of ‘turning on a light’ as a single switch to a single ceiling light, I would agree with you but this type of automation allows for something much more complex: lighting ‘scenes’. This is already done in industrial and some high end home. This goes well beyond a simple light switch. When you then include the ability to have multiple scenes, you can see how a single light switch no longer works.

      If you don’f feel motivated by this scenario, that’s ok. I’m trying to make the broader point that automation allows you to do things that were impossible in the old paradigm. however, even if we are willing to accept the simplistic model of lights, I would *still* argue that this form of automation is worth pursuing as it allows me to have more light, better comfort but with aggressive shutdown of empty rooms, actually save energy. That seems both conservative and worthwhile.

      But clearly (and this was the point of my article) if this system is irritating, it’s not worth any amount of savings. It has to not just save energy (or enable scenes) but also work, reliably, all the time….

      • Boltar

        Actually I think we’re in pretty good agreement. No question efficiency, safety, even disaster preparedness can be advanced with good automation. And not getting in the way, as you point out, will be critical to avoiding consumer backlash.

  • Anthony Townsend

    if its this hard to get bedrooms right… i shudder to think about how HardHard designing entire ecosystems of these things in public settings like city streets, workplaces is going to be.

    • Scott Jenson

      My point wasn’t to scare people away, only appreciate the complexity of the problem. My hope is that we’ll start breaking down tasks into conservative vs risky groups. Conservative tasks would like be conservation driven tasks like turning down the heat when no one is home or turning on low intensive baseboard lights. Risky tasks would be unlocking the door or turning on bright overhead lights. This way we *can* do full automation of the conservative tasks, we just set a higher bar for the risky ones (or as I suggest above, ask for a stronger user intent before actually turning them on.

      Take the door unlock case. Just adding a touch sensor to the door knob makes all the difference. The house can detect my identity outside the door (many ways to do this) but it isn’t until I actually touch the door knob that the lock is opened. This double triggering makes it much clearer that I really do want to enter the room.

      I’m so worried my article is going to be taken the wrong way. I’m very bullish on this issue, it just needs to be done properly!

      • Anthony Townsend

        No, you have done a wonderful service highlighting the need for paying more attention to user experience. So many of the firms in the smart cities space don’t really get consumers/citizens/users and don’t pay enough attention to this stuff.

  • Home automation is that the future! when the line, production and method automation, the ‘new’ reality resides area automation! because the new-age technology integrates into day-after-day life, a lot of individuals need to decide on the intelligent, resilient, secure and responsive answer to create life safer, secure, convenient and connected.