Jud will be speaking at Strata next week on a panel titled “What’s Mine is Yours: the Ethics of Big Data Ownership.”
If you’re attending Strata, you can also find out more about growing business of data marketplaces at a “Data Marketplaces” panel with Ian White of Urban Mapping, Peter Marney of Thomson Reuters, Moe Khosravy of Microsoft, and Dennis Yang of Infochimps.
My interview with Jud follows.
Why is social media data important? What can we do with it or learn from it?
Jud Valeski: Social media today is the first time a reasonably large population has communicated digitally in relative public. The ability to programmatically analyze collective conversation has never really existed. Being able to analyze the collective human consciousness has been the dream of researchers and analysts since day one.
The data itself is important because it can be analyzed to assist in disaster detection and relief. It can be analyzed for profit in an industry that has always struggled to pinpoint how and where to spend money. It can be analyzed to determine financial market viability (stock trading, for example). It can be analyzed to understand community sentiment, which has political ramifications; we all want our voices heard in order to shape public policy.
What are some of the most common or surprising queries run through Gnip?
Jud Valeski: We don’t look at the queries our customers use. One pattern we have seen, however, is that there are some people who try to use the software to siphon as much data as possible out of a given publisher. “More data, more data, more data.” We hear that all the time. But how our customers configure the Gnip software is up to them.
With Gnip, customers can choose the data sources they want not just by site but also by category within the site. Can you tell me more about the options for Twitter, which include Decahose, Halfhose, and Spritzer?
Jud Valeski: We tend to categorize social media sources into three buckets: Volume, Coverage, or Both. Volume streams provide a consumer with a sampled rate of volume (Decahose is 10%, for example, while a full firehose is 100% of some service’s activities). Statisticians and analysts like the Volume stuff.
Coverage streams exist to provide full coverage of a certain set of things (e.g., keywords, or the User Mention Stream for Twitter). Advertisers like Coverage streams because their interests are very targeted. There are some products that fall into both categories, but Volume and Coverage tend to describe the overall view.
For Twitter in particular, we use their algorithm as described on their dev pages, adjusted for each particular volume rate desired.
Gnip is currently the only licensed reseller of the full Twitter firehose. Are there other partnerships coming up?
Jud Valeski: “Currently” is the operative word here. While we’re enjoying the implied exclusivity of the current conditions, we fully expect Twitter to grow its VAR tier to ensure a more competitive marketplace.
From my perspective, Twitter enabling VARs allows them to focus on what is near and dear to their hearts — developer use cases, promoted Tweets, end users, and the display ecosystem — while enabling firms focused on the data-delivery business to distribute underlying data for non-display use. Gnip provides stream enrichments for all of the data that flows through our software. Those enrichments include format and protocol normalization, as well as stream augmentation features such as global URL unwinding. Those value-adds make social media API integration and data leverage much easier than doing a bunch of one-off integrations yourself.
We’re certainly working on other partnerships of this level of significance, but we have nothing to announce at this time.
What do you wish more people understood about data markets and/or the way large datasets can be used?
Jud Valeski: First, data is not free, and there’s always someone out there that wants to buy it. As an end-user, educate yourself with how the content you create using someone else’s service could ultimately be used by the service-provider.
Second, black markets are a real problem, and just because “everyone else is doing it” doesn’t mean it’s okay. As an example, botnet-like distributed IP address polling infrastructure is commonly used to extract more data from a publisher’s service than their API usage terms allow. While perhaps fun to build and run (sometimes), these approaches clearly result in aggregated pools of publisher data that the publisher never intended to promote. Once collected, the aggregated pools of data are sold to data-hungry analytics firms. This results in end-user frustration, in that the content they produced was used in a manner that flagrantly violated the terms under which they signed up. These databases are frequently called out as infringing on privacy.
Everyone loves a good Robin Hood story, and that’s how I’d characterize the overall state of data collection today.
How has real-time data changed the field of customer relationship management (CRM)?
Jud Valeski: CRM firms have a new level of awareness. They no longer rely exclusively on dated user studies. A customer service rep may know about your social life through their dashboard the moment you are connected to them over the phone.
I ultimately see the power of understanding collective consciousness in responding to customer service issues. We haven’t even scratched the surface here. Imagine if Company X reached out to you directly every time you had a problem with their product or service. Proactivity can pay huge dividends. Companies haven’t tapped even 10% of the potential here, and part of that is because they’re not spending enough money in the area yet.
Today, “social” is a checkbox that CRM tools attempt to check off just to keep the boss happy. Tomorrow, social data and metaphors will define the tools outright.
Have you learned anything as a social media user yourself from working on Gnip? Is there anything social media users should be more aware of?
Jud Valeski: Read the terms of service for social media services you’re using before you complain about privacy policies or how and where your data is being used. Unless you are on a private network, your data is treated as public for all to use, see, sell, or buy. Don’t kid yourself. Of course, this brings us all the way back around to black markets. Black markets — and publishers’ generally lackadaisical response to them — cloud these waters.
If you can’t make it to Strata, you can learn more about the architectural challenges of distributing social and location data across the web in real time, and how Gnip has evolved to address those challenges, in Jud’s contribution to “Beautiful Data.”