Additional iPhone tracking research

Researchers and reporters are exploring many of the issues related to mobile location data.

Update, 4/27/11 — Apple has posted a response to questions raised in this report and others.

By Alasdair Allan and Pete Warden

Here’s the latest developments on iPhone tracking.

Android records a short log

The Guardian has a good overview of Android’s equivalent to consolidated.db. It records the last 50 cell locations, and the last 200 Wi-Fi networks, but older entries are overwritten. As we mentioned in our original video, this was what we expected on the iPhone when we found the file, and it was the sheer scale and duration of the recording that floored us, along with how easy it was to access on your computer. Android doesn’t appear to copy the file over when you sync, so you’d need physical access to the phone to read it.

Phoning home your location

In the Wall Street Journal there’s a good story covering how phones often send your location back to servers at both Apple and Google. We’ve known that cell companies are gathering this kind of data, because they need it for their basic operations, but the most interesting question for me is how it’s actually stored by these software companies. If it’s truly just for improving their location services, it could be anonymized so that it would be hard to figure out an individual’s movements if you had the data. Even if it’s not, the data is somewhat protected when it’s on a company’s internal network, since that keeps it further out of reach than a file that’s held on your machine.

Better for tracking travel than home or office locations

Sean Gorman and my friend Peter Batty have done some impressive work digging into the details of the location data. Their conclusion is that it’s hard to spot locations where you spend a lot of time in the same place, like your house or place of work. It’s almost as if re-visiting the same spot overwrites a lot of the older data for that place, which would fit with a lot of what we’ve seen. They also try to quantify the accuracy of the location, pointing out how many outliers appear.

Even just showing where you’ve been traveling to is pretty concerning, but it’s good to rule out some malicious uses. The work they’ve done gives us a lot more about the characteristics of the data, I’m looking forward to seeing more of this kind of analysis.

Intriguingly, their work also has some support for Will Clarke’s idea that the locations are associated with cell towers. Peter’s data shows a cluster around Mile High Stadium, which he hasn’t visited recently but which does have a lot of cell infrastructure. Sean has another map that overlays actual tower locations with his points, and it’s clear they don’t coincide, but could well be triangulated from multiple towers. Sean’s observation fits with our initial hypothesis that the locations are the result of sometimes-inaccurate triangulation from towers, but Peter’s is evidence that there’s a bias in the data to clustering around tower positions.

Peter is investigating the WiFiLocation table. This typically contains a lot more points than the cell version, with 219,000 entries in Alasdair’s data versus only 29,000 cell points. We didn’t visualize this in the application because the derived lat/long points are a lot noisier, but that may be an issue with the quality of the location-lookup tables Apple are using since they switched away from SkyHook. It appears to record the ID of many of the WiFi networks you’ve come into range of, so I’ll be interested to see what Peter and others discover about this data.

Related:

tags: , , ,
  • http://www.google.com Accuracy

    Why lace this with mystery and innuendo. It seems as though you are trying to do maximum damage to Apple’s reputation even as the substance of your earlier accusations evaporates. I’d have expected honest from O’Reilly.

    Apple explained exactly what this information is used for in Session 115 of WWDC 2010. The evidence you are breathlessly uncovering supports exactly what they describe.

    GPS is slow and power hungry. To improve performance and extend battery life, the iPhone downloads locations of cell towers and wifi routers in your vicinity and use these to triangulate your position. Since most of us keep going back to places they have already been, the database means that the phone can determine your location without needing network access.

    This means that location based apps like Maps, or tagging photos can respond immediately without having to burn a lot of power. Who wants to wait 30 seconds for maps to figure out where you are?

    Because the radio environment changes (as people add and remove wifi routers, and buildings are constructed changing the sight lines to cell towers), the phone can’t store a database of all the radios in the world, so it stores the set of radios in the region around the places you go.

    Again, this isn’t secret – Apple has disclosed it. Questioning their motives without doing proper research is dishonest. The unspoken question is what your motives are?

  • Mathieu

    @Accuracy Most iPhone users didn’t know that Apple was tracking their location before this website made the buzz last week.
    The problem is that iPhone users (unlike Android users) can’t disable this feature and stop being track by Apple.
    The other problem (that Apple will fix soon) is that the file containing the location is backed up to the user Mac/PC using iTunes and that anyone accessing this Mac/PC can see the iPhone location for as long as 1 year.
    On Android the file is not backed up to any Mac/PC and the file is not accessible on the phone by anyone except the Google Network Location app (if the device is not rooted).

    The ‘wireless network’ based location is a great feature for smartphones and does require mapping from previous user locations but the user should have the choice to use this service or not.
    On Android, users have the choice … in fact, one the steps of the wizard when they start using their Android device is asking user their permission (as you can see in this article picture http://techcrunch.com/2011/04/22/google-responds-to-smartphone-location-tracking-uproar-says-android-is-opt-in/)

  • http://Http://www.google.com Accuracy

    @Matthieu:

    Apple isn’t tracking user’s locations. If most users think that now, it’s because this site presented a falsehood in a sensationalizing manner. Saying ‘Apple is tracking user’s locations’ is simply not true.

    The file does NOT contain the user’s location. People who access the MAC/PC cannot see the iPhone location for as long as a year.

    It does contain a vague region of cell towers and wifi networks around locations that the user has been, but these are typically miles away from where the user has been, and the timestamps do not reflect when the user was there, nor do they create a ‘track’ of the user’s path because the phone isn’t recording the user’s location.

    The phone is keeping a map of radio’s near where the user goes. That isn’t the same thing as tracking the users location.

    The real world equivalent would be a briefcase containing a city map of every city you had visited. Yes, this would give you some information about a person’s moments, but representing it as ‘tracking” would be laughable.

    Here’s a piece that presents it honestly: http://geothought.blogspot.com/search/label/location

    The bottom line is that Apple hasn’t been tracking anyone’s location, and what they are doing was never secret. Claims to the contrary are either misinformed or dishonest.

  • ath0

    @Accuracy is 100% right and I completely share his opinions.

    These people released incomplete research and painted it in deep colours of of fear, uncertainty and doubt.

    Even now that research has picked up and is showing the histeria behind it all, the authors are still twisting reality by saying things like “data could be anonymised”. For crying out loud, it is anonymised!

    Which is no surprise because Apple had said so to US congress last year. Do you think a company like Apple would risk everything being caught lying to US congress just get private location data for a service they don’t even provide?

    I also find it amusing the authors didn’t show the Wifi data because data was so noisy. How convenient, only show the better data and ignore the rest. Apparently you forgot to mention that noise as well in your initial disclosure.

    I’m really shocked one of these supposed researchers even wrote a book on iPhone programming for ORA. I haven’t seen the book but if this research is any indication it must be one of worst books ever, probably more like iPhone Programming for Dummies.

    ORA has really gone fully down the drain on this one. Enjoy the Web 2.0 hot air and sensationalistic reporting. ORA you are over for me.

  • CaTAC

    Spot on analysis @Accuracy. I also question their motivation on this.

    But to understand why let’s use the same sharp research methods as the authors of this study (ie, speculation):

    Maybe O’Reilly was about to release an app offering subscriptions to their content on Apple’s App Store, but then couldn’t negotiate a lower than 30% commission due the recent enforcement of app store rules.

    Given the background of the authors in iOS development it’s possible they were even involved directly in that project.

    Maybe they were (or still are) hoping that by creating this mass histeria they could use it as leverage in the negotiations.

    Sounds plausible?

  • http://www.amitiae.com Graham Keith Rogers

    I was looking round for background to this and found the Nokia User Agreement which refers to this location data and (among other things) explains how they would share it with law enforcement if needed. This implies retention of the information. http://www.nokia.com/about-nokia/privacy/privacy-policy#s8397158x2

  • http://www.google.com Accuracy

    Graham Keith Rogers – what on earth does Nokia’s user agreement have to do with Apple?

  • http://www.xionsms.com/ smith klevin

    i think most of the cellular companies trace there caller to check there services and area of service it does’nt matter if they are doing it.

    step by step guide to jailbreak your iphone.

    http://www.xionsms.com/2011/04/jailbreak-iphone-4-and-iphone-3gs-using.html

  • Fred Clouse

    i track my iPhone with software called Mobile Spy http://www.mobile-spy.com . i can track GPS and more from a secure web panel.

  • Walt French

    @Accuracy, there’s a more fundamental reason to remind @Mathieu that Apple is not tracking users: it is YOUR PHONE that is logging the location information, and I have yet to see claims that these logs are transmitted to Apple Inc. (“Apple”). The logged information stays under your control — you can erase it, for example — and location info that is sent to Apple apparently is done so under appropriate agreement & disclosure (e.g., to support “find my phone.”)

    I don’t think it’s entirely appropriate for Apple to have set up the software to record this information, apparently ready for abuse and not visibly providing a service to the user. This is a user gotcha and that alone is a good reason for taking Apple to task.

    But it is REALLY inappropriate to say, “Apple is tracking users.”

  • http://www.google.com Accuracy

    @Walt French – for sure – you point out the most egregious problem with what O’Reilly has done – they’ve made an allegation that they know would damage Apple without having any evidence to support it. In some jurisdictions this might well count as libel.

    It’s true that the data could potentially be abused. The potential is however far far lower than O’Reilly has claimed – because it doesn’t contain either your accurate location, or accurate timestamps of when you were there, and is much harder to steal than your email.

    There is a very clear service provided to the user by this data – faster location services (for maps, tagging photos etc) using less power, in more places. Apple explained this a year ago.

    Do a google search for “Friends of O’Reilly”, and you’ll discover that they have close ties with Google.

    Is this evidence that they have deliberately acted to damage Apple on Google’s behalf? No, of course not. But if their intention is not to damage Apple, they should correct the misunderstanding they created by publishing a retraction. E.g. “we were wrong: Apple does not track your location”.

  • ath

    @Accuracy I would understand the ties with Google but one of the authors of this misleading report wrote THE ORA book on iPhone programming. Why is he shooting himself in the foot like this? Surely he should support the platform, not help kill it with mystery and innuendo like you say.

    @CaTAC I wouldn’t put doing that past these people.

    Actually I was surprised to find only ONE iPad app from O’Reilly on the store – for their HTML5 book – and even then by the user review it seems extremely buggy. But the “Friends of O’Reilly” comment from @Accuracy may explain that.

    In other news I do find it amusing that the authors complain Apple does not reply to their claims, yet they too completely fail to reply to any of the comments in their own blog posts.

  • http://dirk.net Dirk

    Isnt’ it conceivable that Apple is generating “location profiles”, i.e. gathering data about every location in terms of cell phone towers, wifi-networks, reception strengths, other iPhones/iPads in the area etc., and that from this data they can easily calculate/triangulate the user’s position?
    In a built-up environment and insided buildings there is no GPS reception, so this algorithm would be more precise and reliable than GPS data.
    The hacker challenge now is to calculate the precise lat/lng from the recorded data to find the algorithm.
    Finally: Why is Apple harvesting this data twice a day as f-secure appear to have found (http://www.f-secure.com/weblog/archives/00002145.html). Is the Skyhook connection and Apple building it’s own *location for advertising* (not cell tower) database completely irrelevant?

  • http://www.google.com Accuracy

    @ath – You would think that people who wrote a book on iPhone programming would have at least looked at Apple’s own materials on the subject. Apple explained what the information they download is for in Session 115 of WWDC last year – available for any registered developer to watch.

    Surely any responsible person about to post something so self-evidently damaging would at least do a cursory check of Apple’s own materials. If they did, then they are flat out lying. Let’s assume they didn’t – which makes them incompetent or negligent.

    Let’s also not pretend that Tim O’Reilly, who funded this piece, doesn’t know what’s been said. It’s front page news and governments are getting involved for god’s sake. He’s ultimately responsible and could publish a retraction, or ask his employees to do so. Where is his integrity in this?

    @Dirk – You are right – Apple is indeed building a database of cell towers and wifi networks in order to triangulate user positions faster and less expensively than GPS, and in places where GPS doesn’t work. They explained this to congress last year. This does not involve tracking individual users.

    By continuing to pretend that Apple is tracking users, and that the reason for the iPhone to download mapping data in the vicinity of the user is unknown, the authors of this piece and O’Reilly media are now deliberately spreading misinformation. The right thing to do would be to retract the statements that have been shown to be false, or provide proof that they are true.

  • James

    I can’t understand the two sided views of the iPhone and Android users. Both Google (all Android)and Apple transmit the same amount of periodic data to the respective company, unless you opt out on ewither phone. The difference is Apple obviously made a mistake by leaving the file to accumulate info and being easy to pull cell phone tower and Wifi data in a log file. Apple uploads this periodically, it appears a couple fortimes a week. Google on the other hand uploads at 50 entries every few hours. Once it is on the Google servers, everyone assumes it is “safe”. I would not assume that at all. If anything the more frequent uploads can be for a variety of reason. Most users never read all of the fine print on either phone for “opting out” of the tracking info. Interestingly you can opt out on either phone. Apple made a mistake by continuing to write the log anyway, though it appears from what I have read that it is not transmitted. So let Apple fix its bug and delete the file frequently. And there already are apps that will erase the data as it is written if you want to jailbreak your phone. And no my wife or mother will not find it easy to use root and do the same thing on Android. In other words you can programatically get around either phones logging and transmitting information. The issue is that Gogole keeps only the fifty last entries and Apple has a bug in the iPhone IOS 4.x that keeps the data forever it appears. They both get the exact same amount of information transmitted to Apple or Google (Android) Use any GPS enabled device at your own risk.

  • prachil tambe

    @pete

    are u mac user?
    or droide user?
    whether u can prove that they are tracking once movement?

    if yes, challenge them ( apple people) that they are tracking the devices and force them to change location tracking feature, and allow owner to turn off the feature trully without any background tracking.

  • Franz

    @James

    As you’ve pointed out, there are 3 key differences between those two platforms:

    A) iOS devices retain a couple orders of magnitude more historical location data.

    B) iOS users cannot opt out of this location data storage, even if they have indicated their intent through an “opt out” setting.

    C) iOS location data is not private to the device, it’s backed up to your PC by iTunes, widening its exposure to compromise.

    Some have suggested that iOS might be downloading location data for nearby cell towers and storing it in that database, but shouldn’t it only be doing that if you’ve opted in? And doesn’t the database continue to accumulate location entries even if you’ve opted out? Besides, it would need to transmit your location in order to receive location-related data, which shouldn’t be allowed if you’re in opt-out mode.

    I wonder what the map looks like when you wander around in opt-out mode?

  • ath0

    @Franz

    I would argue that given there are many so many errors in the data (eg Las Vegas effect, etc) by storing more historical data there amount of “noise” will also be greater.

    That’s what I see in my own data: there are so many outlier points in 8 months of data putting me all over places I haven’t been it makes guessing even the general area I live or work in impossible.

    Sometimes more noise is good.

    You are right on the opt out though, if I turn off Location Services I would expect even local caching to stop. But that’s probably a simple bug – given the many super useful apps that make use of location I don’t think many people will turn it off explicitly, maybe it wasn’t fully tested.

  • Franz

    @ath0

    Looks like the continued caching after opt-out was indeed a bug, according to today’s Q&A from Apple:

    Apple Q&A on Location Data

    In the next iOS update, they’re going to delete and discontinue the cache when you opt out, encrypt the database, reduce its length to 7 days, and stop iTunes from backing it up. Sounds like a reasonable plan of action to me.

  • http://phonetapping.org Dan @ Phone Tapping

    Was there ever any proof that the iPhones were sending the data back to Apple, and was the data being stored?

  • http://www.iphone6releasedate.org iPhone 6

    I’ll be curious to see what kind of phone tracking options are available on the iPhone 6 when it comes out.