Play with your data

I was lucky enough to spend a few hours with Andreas Weigend recently, head of the Stanford Social Data lab. He has nine rules of data, and the first is “Start with the problem, not the data.” What struck me about visualizations is that I actually take the opposite approach. I find the only way to begin is to explore what information is available and get a feeling for what stories it can tell.

In my case, we have a Cassandra cluster with information on more than 350 million photos shared on Facebook. I’ve been running Pig analytics jobs regularly to get a view of what we have in there. One of the reports we generate is a count of how many photos and users we have for particular places:

Click to enlarge.

I was chatting with my colleague Chris Raynor about this, and he asked me if we could tell where all the visitors to those places were coming from. This was something that had been at the back of my mind for a long time. Seeing how much information we had on each destination made me realize we had enough data to produce significant and meaningful answers.

When I was learning engineering, one of my favorite case studies was an investigation into an air-traffic control system. Software engineers couldn’t understand why fully-computerized control rooms were actually less efficient and safe than more old-fashioned sites. What the researchers discovered was that the old process of passing around and arranging small cards that each represented a plane gave controllers a much stronger awareness of the situation than a screen that didn’t require their involvement for tasks, such as handing an aircraft to a colleague. I think the same is true of data. The more time you spend manipulating and examining the raw information, the more you understand it at a deep level. Knowing your data is the essential starting point for any visualization.

Pick a question

Now that I had a rough idea for what I wanted to visualize, I really needed to focus on what I would be doing. The best way to do that is to chose the exact title you want to give your visualization. I actually messed this up on one early map I created, giving the blog post the title “How to split up the US.” Everyone subsequently described it as “The Five Nations of Facebook.” Since then, I’ve tried very hard to pick the most natural title for what I’m going to be presenting, and then ensure I can deliver on the promise of the headline.

In this case I had a clear idea of the question at the start, it was going to be “Where do people go on vacation?”. However, as I thought about it, I realized it needed to be a lot more specific and concrete. There’s already a lot of “top travel destinations” lists out there, so what made mine different? It was the use of Facebook to gather much richer and more detailed information, so I refined it to “Where do Facebook users go on vacation?”.

Sketch out your presentation

I now had the data and a question I wanted to answer. The next step was figuring out how to show the information in a visual form. I’m in love with network diagrams showing connections between thousands of objects, but so often they are completely baffling to the rest of the world. I still remember David Cohen threatening to strangle me if I showed him another one of “those damn spider webs” instead of a business plan. However, network diagrams are a good way of hinting at how much data is available for querying; they can really give an idea of the sheer scale of what’s there.

One of my favorite recent visualizations was Paul Butler’s map of friendships on Facebook, so I decided to use that as a visual reference:

See the full version of Paul Butler’s “Visualizing Friendships” visualization.

I borrowed a couple of key ideas from his work: the general color palette of the blue lines on a dark background and the use of great circles to create flowing arcs for all connections.

As I thought about the presentation, I realized that I had to simplify what it would be showing. With sources and destinations plotted all over the world, both the visual look and the querying interface would be overwhelming. Our user-base is primarily American thanks to our reliance on English-only natural language processing, so with that in mind I decided to make life simpler by only showing data from people who lived in the U.S. Accordingly, I changed the question in my title to “Where do American Facebook users go on vacation?”.

While I’m mostly presenting this as a linear, waterfall process, what I’ve just described is a good example of how iterative cycles drive the real workflow. It’s hard to know how well a lot of things will work until you try them. As you’re still making some progress, don’t worry if you find yourself going in circles.

Crunch the data

If you know your data, and you have a good idea of the question you’re trying to answer, this should be the simplest stage. You’ll hopefully have a clear set of requirements and it’s just a matter of executing the right queries over your data.

In this case I already had some Pig scripts asking similar questions, so I was able to adapt one of those. The biggest surprise was when I ran into issues with some of the joins. The hard part was running the Hadoop job to gather the raw data from our Cassandra cluster, and that worked. I was able to output smaller files containing the gathered data, and then run a local Pig job to do the joins I needed.

The next stage was turning the raw information into a form that could be displayed. For example, I needed to take all of the user locations from the unstructured text strings that Facebook gave me, and convert them into latitude-longitude coordinates for plotting on a map. For this sort of work I usually turn to a general-purpose scripting language, and most of Jetpac is already written in Ruby, so that was an easy choice. I wrote a script that walked through the data, using the Data Science Toolkit to match coordinates with names, and then output it into a file containing a JSON array of all the information.

Build an interface

A lot of the best visualizations have no interactivity. They just tell a story with a static image. That’s why it’s worth considering whether you need an interface at all. I actually had the interactive site that I used to create the “Five Nations of Facebook” visualization up for several weeks before that post, and nobody used it because it was too confusing. It was only when I boiled it down into a single picture with labels that it became a hit.

My problem is that I want other people to have as much fun exploring the data as I’ve had, so I couldn’t resist adding some interaction to the vacation visualization. I still wanted to retain the immediate visual appeal of a static image, so I decided to create a background showing the full data to introduce the visualization at a first glance, and then overlay an interactive foreground once the user started exploring it more deeply.

In most cases you’re better off using one of the excellent off-the-shelf visualization frameworks like D3. Since I needed something client-side for interaction, and was working with both geographic and network rendering, I couldn’t find anything that met my requirements. Instead I cannibalized one of my own projects, the jQuery component from OpenHeatMap, and combined it with HTML5 canvas rendering to produce a custom JavaScript renderer. I used it to pre-render a background containing all the possible connections between home towns and travel destinations, and saved that off as a static image. That’s useful to save rendering time on page load, and lets me fall back to a static visualization on older browsers that don’t support Canvas.

Click to enlarge.

I then tied in rendering the connections of any places that the user was hovering their cursor over, so that they could quickly get a feel for the relationships expressed in the data. I also wanted to display the details underlying the picture, so to drill down I added a dialog listing the raw statistics about a place. Users can bring this dialog up by clicking.

Click to enlarge.

One problem with that interaction is that a lot of different cities are in a very small area, so it becomes extremely difficult to pick the one you want with the mouse cursor. To make that a little better, I prioritized the most popular U.S. cities so that in case of a conflict, they’re chosen over their smaller neighbors. I realized I also needed to add a search box. Thankfully we’re heavy users of Twitter’s Bootstrap framework, so it was a simple matter to add a search field and tie it in with Twitter’s excellent autocomplete component.

Find the surprises!

I build these visualizations so I can explore them myself, so my favorite part of the whole process is the chance to sit and play with the results. There’s always unexpected stories hidden in there, and I love uncovering them. For example, who knew that the city that had the most visitors to Paris was West Hollywood? When I lived in Los Angeles I used to love popping by the wonderful patisseries. Now I know why they’re so good! These little details are the stories that catch people’s imagination and cause them to spread the word, so think about writing a few of them up to help visitors understand what the page can tell them.

You’ll never know whether one of your visualizations will become popular ahead of time, but the real reward is enjoying your own work. I hope this short guide gives you some ideas for visualizations you want to build. I look forward to seeing what you come up with.

See the full visualization.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

Related:

• Visualization deconstructed: Mapping Facebook’s friendships
• Visualization deconstructed: New York Times “Mapping America”
• Visualization deconstructed: Why animated geospatial data works

People are fascinated by mirrors

There’s a good chance you’ve used their Twitter Grader tool, and its popularity shows one of the secrets to HubSpot’s success. The inspiration for the company came when Dharmesh realized that his own blog was driving a lot of traffic, and the startups he was helping out were all struggling to get anywhere near the same number of visitors. He built HubSpot by applying what he’d learned from blogging, and one of the key lessons was that people crave new information about their own lives and projects. If you can create a service that gives people interesting data about themselves and their organizations, they’ll spend time exploring it and they’ll share it with their friends.

With Twitter Grader, Dharmesh didn’t just create a source of free advertising for his company, it’s also implicitly targeted at people who want to improve their presence on the social network. Many of these people will be the small business owners that are in his target market. Even better, by offering the statistics as a gift to users, he created a small sense of reciprocal obligation that will make them more likely to purchase his services. The approach started with their original Website Grader service, but they found it so powerful, HubSpot now has a whole range of similar free tools for analyzing everything from your Facebook page to your blog.

The lesson for me is that giving people data and visualizations about things they truly care about can be a powerful tool for drawing them in to your service. Do some creative thinking about your customer’s problems, and see if there’s something you can offer them as a reward for their attention.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

You should kill unicorns and rainbows with science

One of the most enjoyable conversations I had at HubSpot was with Dan Zarrella, who describes himself as a social media scientist. I can already hear some physics PhDs grinding their teeth, but Dan has earned that title by applying a lot of much-needed rigor to the fluffy world of social media measurements. He’s crusading against “unicorns and rainbows” metrics that have no connection to the goals you want to achieve. Many businesses have focused on building up easy-to-measure numbers like fan or follower counts, but to use Eric Ries’ term, those are just vanity metrics. You can gain a million friends without it leading to a penny in revenue.

Dan’s antidote is the relentless application of logic and analysis, working backwards from the business goals to evaluate everything you’re doing as objectively as possible. A fantastic example of this is his study looking at how minor content details, like punctuation, make a retweet more or less likely. It’s possible to argue with particular conclusions he draws, but he’s transparently laid out the methods by which he arrived at them. Anybody with some technical knowledge and access to a decent chunk of Twitter data can try to reproduce and refine his results. This makes the report so much more useful than the opinions or impressions that dominate most discussions of social media, since we can actually have an evidence-based argument about it.

I came away from talking with Dan with a new appreciation of how powerful the scientific method can be in even the most unlikely situations. I’ll be taking a fresh look at some of the painful problems my projects are hitting, and seeing if there’s some way I can gather the right data to gain insights, even if they seem hopelessly qualitative at first glance.

User education is painful but powerful

HubSpot focuses on the sort of people who used to buy ads in the Yellow Pages to promote their businesses. These people know they now need to use the Internet to reach customers, but they aren’t sure how. To succeed, HubSpot has to help those people build useful websites and channels. Templates and other automated tools help, but a lot still comes down to people creating the right content for their own businesses and responding appropriately when customers get in touch through Twitter, Facebook or email. The only way to achieve that is to teach people how to do it, and so a lot of the company’s resources are put into education.

On a simple level, tools like HubSpot’s graders offer simple suggestions for improving websites and other content. Users of the service are sent regular emails that remind them of steps and actions they need to take, such as updating their blogs. HubSpot hosts a popular video cast that covers all sorts of tips and horror stories from the last week in social media. All of these efforts really seem to help the company, judging from how enthusiastically users respond to all the material. On a deeper level, it also seems to help build a long-term relationship between the company and its customers, driving real loyalty.

One of the unwritten rules of the consumer technology world is that anything that requires educating users is a losing proposition. Anybody who has looked at their customer acquisition funnel knows how even minor usability problems can drive away vast swaths of people. What’s different about HubSpot is that their customers are a lot more motivated than your average consumer on the web. They’re using the service in the hope of actually making more money, so they’re willing to invest some time. It left me wondering if I should spend more time creating training material for my own projects, rather than always prioritizing interface work to make them easier to use. The people who use them to create content are already investing their own time, so is that perhaps another situation where education would pay off?

Hubspot is a smart, practical company that’s very focused on using the data they’re gathering to understand what their customers really need. Maybe that’s precisely because the team isn’t in the Valley to be distracted by every shiny new idea? No matter what the cause, I’m grateful that they spent the time to show me what they’d learned, and I’m looking forward to applying these ideas to my own work.

Related:

• The Facebook Marketing Book (Book)
• The Social Media Marketing Book (Book)
• HubSpot’s Dharmesh Shah discusses inbound marketing at Web 2.0 Expo SF 2010
• Pages before ads and other Facebook marketing tips
• Some of the best tools for Twitter statistics

Take tides for example. You’ve probably never worried about the timing or height of the sea, but for Victorian sailors figuring out the tides was a life or death problem. Getting it wrong would mean a slipped schedule at best, or a shipwreck at worst. The only people who could accurately predict the tides were harbor masters, since conditions varied widely across different areas and required patient observation by locals. The harbor masters guarded their knowledge so carefully that even British naval captains had to pay them to get access to the information they needed to dock their vessels!

The harbor masters were data producers with a business model that excluded many potential users because the transaction costs were too high to be worthwhile. Sound familiar? That’s the state of many of the datasets I wish were openly available, from real-estate listings to full zip-code boundaries.

The Victorian solution was another familiar face — crowdsourcing. William Whewell arranged for hundreds of volunteers around the world to measure their local sea levels and send the numbers back to him. He then plotted the times of the tidal maximums on a map to create a visualization called a co-tidal chart. Below is a modern version from NASA:

Maps like these, along with more detailed tables, allowed navigators to make their journeys without being ambushed by the tides. This story could be a poster child for our own revolution, with open data fixing a painful real-world problem.

The limits of data

What’s really useful about historical analogies is that you can see how they played out in the long term. The villains of the tidal story were the harbor masters who hoarded their information, but in fact that was only a small part of the value they offered. Despite incredibly detailed maps of every port, we still rely on their descendants to pilot commercial ships into harbor. There’s a world of knowledge about currents, shifting sand banks and traffic patterns that it hasn’t been possible to compress into numbers or rules.

The lesson I draw from this is that in many new areas there’s some problems that are easy to fix by gathering and applying data, but we need to keep a bit of humility. James Scott’s “Seeing Like a State” looks at the legacy of the Victorian scientific revolution, and shows how the very success of its ideas had a dark side. Creating datasets may help technical people like us to understand problems and propose solutions, but it also means that harbor masters and other people with deep, lived experience of the domains will be overruled. In the 20th century the prestige of the scientific toolkit was used to justify disasters like the collectivization of agriculture, as technocrats around the world wielded numbers to take power away from “inefficient” smallholders. Those figures were mostly proven bogus by reality, as plans with no knowledge of conditions on the ground failed when confronted with the wildly variable conditions of soil, weather and pests that farmers had spent a lifetime learning to cope with.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

The way forward

Specialists like us who can understand and interpret data are in a privileged position. Most people have an exaggerated respect for arguments expressed as numbers or visualizations, because they don’t understand how many assumptions and simplifications go into these creations. It’s our job to remember that and balance our enthusiasm about the power of our techniques with some humility about their limits. It also makes education and popularization even more important, since we need a common language to talk with domain specialists, so they can keep our work honest with their own deep knowledge. The Victorian example shows that if we’re going to improve the world with data, it’s absolutely essential we stay grounded in reality.

Photo: SteamPunk Frankenstein – By D. Mattocks by SteamPunk Frankenstein, on Flickr

Related:

• Why the term “data science” is flawed but useful
• Data and mathematical intimidation

Something that my friend Arvind Narayanan has taught me, both with theoretical papers and repeated practical demonstrations, is that this anonymization process is an illusion. Precisely because there are now so many different public datasets to cross-reference, any set of records with a non-trivial amount of information on someone’s actions has a good chance of matching identifiable public records. Arvind first demonstrated this when he and his fellow researcher took the “anonymous” dataset released as part of the first Netflix prize, and demonstrated how he could correlate the movie rentals listed with public IMDB reviews. That let them identify some named individuals, and then gave access to their complete rental histories. More recently, he and his collaborators used the same approach to win a Kaggle contest by matching the topography of the anonymized and a publicly crawled version of the social connections on Flickr. They were able to take two partial social graphs, and like piecing together a jigsaw puzzle, figure out fragments that matched and represented the same users in both.

All the known examples of this type of identification are from the research world — no commercial or malicious uses have yet come to light — but they prove that anonymization is not an absolute protection. In fact, it creates a false sense of security. Any dataset that has enough information on people to be interesting to researchers also has enough information to be de-anonymized. This is important because I want to see our tools applied to problems that really matter in areas like health and crime. This means releasing detailed datasets on those areas to researchers, and those are bound to contain data more sensitive than movie rentals or photo logs. If just one of those sets is de-anonymized and causes a user backlash, we’ll lose access to all of them.

So, what should we do? Accepting that anonymization is not a complete solution doesn’t mean giving up, it just means we have to be smarter about our data releases. Below I outline four suggestions.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

Keep the anonymization

Just because it’s not totally reliable, don’t stop stripping out PII. It’s a good first step, and makes the reconstruction process much harder for any attacker.

Acknowledge there’s a risk of de-anonymization

Don’t make false promises to users about how anonymous their data is. Make the case to them that you’re minimizing the risk and possible harm of any data leaks, sell them on the benefits (either for themselves or the wider world) and get their permission to go ahead. This is a painful slog, but the more organizations that take this approach, the easier it will be. A great model is Reddit, which asked their users to opt-in to sharing their data. They got a great response.

Limit the detail

Look at the records you’re getting ready to open up to the world, and imagine that they can be linked back to named people. Are there parts of it that are more sensitive than others, and maybe less important to the sort of applications you have in mind? Can you aggregate multiple people together into cohorts that represent the average behavior of small groups?

Learn from the experts

There’s many decades of experience of dealing with highly sensitive and personal data in sociology and economics departments across the globe. They’ve developedtechniques that could prove useful to the emerging community of data scientists, such as subtle distortions of the information to prevent identification of individuals, or even the sort of locked-down clean-room conditions that are required to access detailed IRS data.

There’s so much good that can be accomplished using open datasets, it would be a tragedy if we let this slip through our fingers with preventable errors. With a bit of care up front, and an acknowledgement of the challenges we face, I really believe we can deliver concrete benefits without destroying people’s privacy.

Related:

• Why the term “data science” is flawed but useful
• The iPhone tracking story, one week later
• Open question: How much location information are you willing to share?
• Online privacy debates heat up in Washington

With that in mind, here are the arguments I’ve heard against the term, and why I don’t think they should stop its adoption.

It’s not a real science

I just finished reading “The Philosophical Breakfast Club,” the story of four Victorian friends who created the modern structure of science, as well as inventing the word “scientist.” I grew up with the idea that physics, chemistry and biology were the only real sciences and every other subject using the term was just stealing their clothes (“Anything that needs science in the name is not a real science”). The book shows that from the beginning the label was never restricted to just the hard experimental sciences. It was chosen to promote a disciplined approach to reasoning that relied on data rather than the poorly-supported logical deductions many contemporaries favored. Data science fits comfortably in this more open tradition.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

It’s an unnecessary label

To me, it’s obvious that there has been a massive change in the landscape over the last few years. Data and the tools to process it are suddenly abundant and cheap. Thousands of people are exploiting this change, making things that would have been impossible or impractical before now, using a whole new set of techniques. We need a term to describe this movement, so we can create job ads, conferences, training and books that reach the right people. Those goals might sound very mundane, but without an agreed-upon term we just can’t communicate.

The name doesn’t even make sense

As a friend said, “show me a science that doesn’t involve data.” I hate the name myself, but I also know it could be a lot worse. Just look at other fields that suffer under terms like “new archaeology” (now more than 50 years old) or “modernist art” (pushing a century). I learned from teenage bands that the naming process is the most divisive part of any new venture, so my philosophy has always been to take the name you’re given, and rely on time and hard work to give it the right associations. Apple and Microsoft (née Micro-soft) are terrible startup names by any objective measure, but they’ve earned their mindshare. People are calling what we’re doing “data science,” so lets accept that and focus on moving the subject forward.

There’s no definition

This is probably the deepest objection, and the one with the most teeth. There is no widely accepted boundary for what’s inside and outside of data science’s scope. Is it just a faddish rebranding of statistics? I don’t think so, but I also don’t have a full definition. I believe that the recent abundance of data has sparked something new in the world, and when I look around I see people with shared characteristics who don’t fit into traditional categories. These people tend to work beyond the narrow specialties that dominate the corporate and institutional world, handling everything from finding the data, processing it at scale, visualizing it and writing it up as a story. They also seem to start by looking at what the data can tell them, and then picking interesting threads to follow, rather than the traditional scientist’s approach of choosing the problem first and then finding data to shed light on it. I don’t know what the eventual consensus will be on the limits of data science, but we’re starting to see some outlines emerge.

Time for the community to rally

I’m betting a lot on the persistence of the term. If I’m wrong the Data Science Toolkit will end up sounding as dated as “surfing the information super-highway.” I think data science, as a phrase, is here to stay though, whether we like it or not. That means we as a community can either step up and steer its future, or let others exploit its current name recognition and dilute it beyond usefulness. If we don’t rally around a workable definition to replace the current vagueness, we’ll have lost a powerful tool for explaining our work.

Related:

• What is data science?
• 3 skills a data scientist needs
• Becoming a data scientist
• Data science democratized

It’s now been a week since we published the iPhone tracking story, so it seemed a good time to cover what we’ve learned.

The fix

Apple has just released a Q&A covering this problem and they will be fixing the issues we spotted with a software update. “The reason the iPhone stores so much data is a bug we uncovered,” Apple notes in the statement.

Apple explains that nearby locations are pulled down from an Apple database and stored on the phone. These locations are from a “crowd-sourced database of Wi-Fi hotspot and cell tower data.” This matches the picture that was emerging from research. It explains why there’s lots of locations that don’t match towers, and also why the accuracy is within a few-hundred meters, since we’ve learned that “micro-cells” in urban areas are clustered closely together.

The Q&A explains the technical workings behind the log and reassures us that only anonymous data is sent back. Our conclusions still apply.

Apple doesn’t address our claim that this reveals sensitive information about your travels. At this point we’re just relieved to get an explanation and a fix, but people can examine their own data and decide for themselves how happy they would be sharing it with strangers.

Forensics

What Does Your iPhone Know About You? More Than You Think — Alexis Madrigal has written a fascinating follow-up piece covering the data that professionals can read from your phone. Using forensics tools like the Lantern program that Alex Levinson helped build, anyone with physical access to the device can construct a picture of the user’s life. It’s eye-opening what the “law enforcement, government, and corporate examiners” who purchase the system can uncover about your behavior.

The Tell-all telephone visualization also makes for thoughtful viewing. It’s built from details that a German politician forced his cell phone provider to share after it was caught storing six months of location data on its subscribers. I think one of the reasons that the iPhone Tracker application has had so much use is that it shows people their own data in an understandable way. Unfortunately, that means that similar information that’s harder to access behind a company’s firewall may not get the same scrutiny, just because it’s harder to show in a way that connects with people.

Uses for good

I’ve long been a fan of Geoloqi’s opt-in service for recording and sharing your travels, but several other projects in the same area have appeared in my inbox over the last few days. Maria Scileppi has created the Living Brushstroke project (see video below) to capture people’s movements at events, and turn the data into art. Intriguing and beautiful patterns emerge as people cross paths. It’s a very fresh way to look at our lives.

Related:

• Apple Q&A on Location Data
• Got an iPhone or 3G iPad? Apple is recording your moves
• iPhone tracking: The day after
• Additional iPhone tracking research

By Alasdair Allan and Pete Warden

Here’s the latest developments on iPhone tracking.

Android records a short log

The Guardian has a good overview of Android’s equivalent to consolidated.db. It records the last 50 cell locations, and the last 200 Wi-Fi networks, but older entries are overwritten. As we mentioned in our original video, this was what we expected on the iPhone when we found the file, and it was the sheer scale and duration of the recording that floored us, along with how easy it was to access on your computer. Android doesn’t appear to copy the file over when you sync, so you’d need physical access to the phone to read it.

Phoning home your location

In the Wall Street Journal there’s a good story covering how phones often send your location back to servers at both Apple and Google. We’ve known that cell companies are gathering this kind of data, because they need it for their basic operations, but the most interesting question for me is how it’s actually stored by these software companies. If it’s truly just for improving their location services, it could be anonymized so that it would be hard to figure out an individual’s movements if you had the data. Even if it’s not, the data is somewhat protected when it’s on a company’s internal network, since that keeps it further out of reach than a file that’s held on your machine.

Better for tracking travel than home or office locations

Sean Gorman and my friend Peter Batty have done some impressive work digging into the details of the location data. Their conclusion is that it’s hard to spot locations where you spend a lot of time in the same place, like your house or place of work. It’s almost as if re-visiting the same spot overwrites a lot of the older data for that place, which would fit with a lot of what we’ve seen. They also try to quantify the accuracy of the location, pointing out how many outliers appear.

Even just showing where you’ve been traveling to is pretty concerning, but it’s good to rule out some malicious uses. The work they’ve done gives us a lot more about the characteristics of the data, I’m looking forward to seeing more of this kind of analysis.

Intriguingly, their work also has some support for Will Clarke’s idea that the locations are associated with cell towers. Peter’s data shows a cluster around Mile High Stadium, which he hasn’t visited recently but which does have a lot of cell infrastructure. Sean has another map that overlays actual tower locations with his points, and it’s clear they don’t coincide, but could well be triangulated from multiple towers. Sean’s observation fits with our initial hypothesis that the locations are the result of sometimes-inaccurate triangulation from towers, but Peter’s is evidence that there’s a bias in the data to clustering around tower positions.

Peter is investigating the WiFiLocation table. This typically contains a lot more points than the cell version, with 219,000 entries in Alasdair’s data versus only 29,000 cell points. We didn’t visualize this in the application because the derived lat/long points are a lot noisier, but that may be an issue with the quality of the location-lookup tables Apple are using since they switched away from SkyHook. It appears to record the ID of many of the WiFi networks you’ve come into range of, so I’ll be interested to see what Peter and others discover about this data.

Related:

• Got an iPhone or 3G iPad? Apple is recording your moves
• iPhone tracking: The day after

By Alasdair Allan and Pete Warden

I don’t think either of us were expecting to see this story strike such a nerve. There’s been some amazing detective work from researchers across the web, and so here’s a selection of the most interesting immediate reactions.

Alex Levinson — Right from launch, we had an FAQ pointing to articles by people like Ryan Neal and Paul Courbis who had found this file (consolidated.db) before, but hadn’t understood or been able to communicate its significance. The main reason we went public with this was exactly because it already seemed to be an open secret among people who make their living doing forensic phone analysis, but not among the general public — even pretty geeky people like Alasdair and me. We were freaked out by the implications of this data and how unprotected it was, but most of the forensics community seemed to miss quite how creepy ordinary people would find it.

I do appreciate how frustrating this must be for Alex though, and would like to apologize personally to him that we didn’t include his article among the prior research we cited. Unlike the others, it didn’t show up in web searches or the books we referenced. It also didn’t help that most of the follow-up articles by other people left out the details that we’d tried to make clear about who found it first. We obviously didn’t communicate it as well as we thought we had, which is completely our fault.

My Life According to the iPhone’s Secret Tracking Log — Alexis Madrigal has a far more interesting life than me, judging by his map. I especially like the points from a flight with Jim Fallows somewhere over West Virginia. As he says, this data can be incredibly interesting, and as data geeks we were just as fascinated as he is. I actually look forward to a future where we can use this sort of information, but with the user’s permission.

Apple is not “recording your moves” — Both of us have been following Will Clarke’s blog for a while and we liked this article. It’s good to look skeptically at the accuracy of the data both in space and time. We do disagree about one of the conclusions though: that the points are just the locations of cell towers. That was one of our first thoughts when we saw the data. But the fact that there’s thousands of different points scattered across small areas, all in slightly different places, seems like pretty strong evidence that they’re not just the locations of cell towers. Another way of putting that is that there’s a lot more points than there are towers. There’s also lots of points with the same tower ID code that are in different locations. That all led to our conclusion that it was trying to figure out the device’s position, even if it wasn’t very good at it.

Until we get a deeper analysis, that’s just a provisional conclusion of course. But getting smart folks like Will to dig into this and correct anything we’ve got wrong is exactly why we open-sourced it. He also picks up on the Las Vegas Anomaly. Multiple people have reported seeing a phantom trip to the city show up, and one theory (other than a lot of lost weekends) is that Apple has an unpacking or testing facility there. Alasdair’s phone that was shipped with iOS4 shows this, whereas my older device that originally had iOS3 doesn’t, which was suggestive. I wonder if Will’s device is a newer one, too?

OpenStreetMap — The application we released relies on this volunteer-run site to render the background map tiles. We ended up tripling their usual load, according to a team member. They actually fired up extra servers to cope, so I made sure to add a link to their donation page from our main site. If you got something out of the application, please do consider giving something to them, or even getting involved. It’s a fantastic team and community. How many other organizations would have responded to heavy usage by a free client by paying for more servers themselves? I even messed up their credit text on the initial version of the application, but they were very understanding about that too.

Related:

• Got an iPhone or 3G iPad? Apple is recording your moves

When I explain to people why the Big Data movement is important — why it’s a real change instead of a fad — I point to price as the fundamental difference between the old and new worlds. Until a few years ago, the state of the art for doing meaningful analysis of multi-gigabyte data sets was the data warehouse. These custom systems were very capable, but could easily cost millions of dollars. Today I can hire a hundred machine Hadoop cluster from Amazon for just $10 an hour, and process thousands of gigabytes a day.

This represents a massive discontinuity in price, and it’s why Big Data is so disruptive. Suddenly we can imagine a group of kids in their garage building Google-scale systems practically on pocket money. While the drop in the cost of data storage and transmission has been less dramatic, it has followed a steady downward trend over the decades. Now that processing has become cheap too, a whole universe of poverty-stricken hackers, academics, makers, reporters, and startups can do interesting things with massive data sets.

Why does this have to do with CrunchBase? The reporter had some implicit assumptions about the cost of the data collection process. He argued that it required extra effort from the journalists to create the additional value captured in the database. To paraphrase him: “It’s time they’d rather spend at home playing with their kids, and so we’ll end up compensating them for their work if we want them to continue producing it.” What I felt was missing from this is that CrunchBase might actually be just a side-effect of work they’d be doing even if it wasn’t released for public consumption.

Many news organizations are taking advantage of the dropping cost of data handling by heavily automating their news-gathering and publishing workflows. This can be as simple as Google Alerts or large collections of RSS feeds to scan, using scraping tools to gather public web data, and there’s a myriad of other information-processing techniques out there. Internally there’s a need to keep track of the results of manual or automated research, and so the most advanced organizations are using some kind of structured database to capture the information for future use.

That means that that the only extra effort required to release something like CrunchBase is publishing it to the web. Assuming that there’s some benefits to doing so (that TechCrunch’s reputation as the site-of-record for technology company news is enhanced, for example) and that there’s multiple companies with the data available, then the low cost of the release will mean it makes sense to give it away.

I actually don’t know if all these assumptions are true, CrunchBase’s approach may not be sustainable, but I hope it illustrates how a truly radical change in price can upset the traditional rules. Even on a competitive, commercial, free-market playing field it sometimes makes sense to behave in ways that appear hopelessly altruistic. We’ve seen this play out with open-source software. I expect to see pricing forces do something similar to open up more and more sources of data.

I’m usually the contrarian guy in the room arguing that information wants to be paid, so I don’t actually believe (as Lewis Strauss famously said about electricity) all data will be too cheap to meter. Instead I’m hoping we’ll head toward a world where producers of information are paid for adding real value. Too many “premium” data sets are collated and merged from other computerized sources, and that process should be increasingly automatic, and so increasingly cheap. Give me a raw CrunchBase culled from press releases and filings for free, then charge me for your informed opinion on how likely the companies are to pay their bills if I extend them credit. Just as free, open-source software has served as the foundation for some very lucrative businesses, the new world of free public data will trigger a flood of innovations that will end up generating value in ways we can’t foresee, and that we’ll be happy to pay for.

There’s been a revolution in data over the last few years, driven by an astonishing drop in the price of gathering and analyzing massive amounts of information. It only cost me $120 to gather, analyze and visualize 220 million public Facebook profiles, and you can use 80legs to download a million web pages for just $2.20. Those are just two examples.

The technology is also getting easier to use. Companies like Extractiv and Needlebase are creating point-and-click tools for gathering data from almost any site on the web, and every other stage of the analysis process is getting radically simpler too.

What does this mean for journalists? You no longer have to be a technical specialist to find exciting, convincing and surprising data for your stories. For example, the following four services all easily reveal underlying data about web pages and domains.

WHOIS

Many of you will already be familiar with WHOIS, but it’s so useful for research it’s still worth pointing out. If you go to this site (or just type “whois www.example.com” in Terminal.app on a Mac) you can get the basic registration information for any website. In recent years, some owners have chosen “private” registration, which hides their details from view, but in many cases you’ll see a name, address, email and phone number for the person who registered the site.

You can also enter numerical IP addresses here and get data on the organization or individual that owns that server. This is especially handy when you’re trying to track down more information on an abusive or malicious user of a service, since most websites record an IP address for everyone who accesses them

Strata: Making Data Work, being held Feb. 1-3, 2011 in Santa Clara, Calif., will focus on the business and practice of data. The conference will provide three days of training, breakout sessions, and plenary discussions — along with an Executive Summit, a Sponsor Pavilion, and other events showcasing the new data ecosystem.

Save 30% off registration with the code STR11RAD

Blekko

The newest search engine in town, one of Blekko’s selling points is the richness of the data it offers. If you type in a domain name followed by /seo, you’ll receive a page of statistics on that URL

The first tab shows other sites that are linking to the current domain, in popularity order. This can be extremely useful when you’re trying to understand what coverage a site is receiving, and if you want to understand why it’s ranking highly in Google’s search results, since they’re based on those inbound links. Inclusion of this information would have been an interesting addition to the recent DecorMyEyes story, for example.

The other handy tab is “Crawl stats,” especially the “Cohosted with” section:

This tells you which other websites are running from the same machine. It’s common for scammers and spammers to astroturf their way toward legitimacy by building multiple sites that review and link to each other. They look like independent domains, and may even have different registration details, but often they’ll actually live on the same server because that’s a lot cheaper. These statistics give you an insight into the hidden business structure of shady operators.

bit.ly

I always turn to bit.ly when I want to know how people are sharing a particular link. To use it, enter the URL you’re interested in:

Then click on the ‘Info Page+’ link:

That takes you to the full statistics page (though you may need to choose “aggregate bit.ly link” first if you’re signed in to the service).

This will give you an idea of how popular the page is, including activity on Facebook and Twitter. Below that you’ll see public conversations about the link provided by backtype.com.

I find this combination of traffic data and conversations very helpful when I’m trying to understand why a site or page is popular, and who exactly its fans are. For example, it provided me with strong evidence that the prevailing narrative about grassroots sharing and Sarah Palin was wrong.

[Disclosure: O'Reilly AlphaTech Ventures is an investor in bit.ly.]

Compete

By surveying a cross-section of American consumers, Compete builds up detailed usage statistics for most websites, and they make some basic details freely available.

Choose the “Site Profile” tab and enter a domain:

You’ll then see a graph of the site’s traffic over the last year, together with figures for how many people visited, and how often.

Since they’re based on surveys, Compete’s numbers are only approximate. Nonetheless, I’ve found them reasonably accurate when I’ve been able to compare them against internal analytics.

Compete’s stats are a good source when comparing two sites. While the absolute numbers may be off for both sites, Compete still offers a decent representation of the sites’ relative difference in popularity.

One caveat: Compete only surveys U.S. consumers, so the data will be poor for predominantly international sites.

Additional data resources and tools are discussed in Pete’s free ebook.

Related:

• The growing importance of data journalism
• Data science democratized
• What is data science?