The three feet of polluted water that flooded Eyebeam’s work and exhibit space on the West Side of Manhattan during the storm had damaged lots of equipment and soaked much of their archive material. Cleaning whatever could be saved and making a priority list for replacing what was lost were the two main challenges of recovery.

Thanks to generous contributions from philanthropic foundations and private companies — such as the Jerome Foundation, the Andy Warhol Foundation for the Visual Arts, the Andrew W. Mellon Foundation, the Art Dealers Association of America, Time-Warner, and O’Reilly Media — as well as donations from individuals, about half the equipment losses have been covered.

There are still some other funding proposals under consideration, but essentially the equipment recovery process has been one of triage: Eyebeam has tried to replace equipment needed immediately by their artists in residence, as well as some practical pieces like the two scissor lifts required to access lights and other equipment at the top of their two-story exhibit space.

After replacing some of the equipment, such as computers and monitors, that artists needed right away to do their work, the next priority was to replace the tools needed to put on exhibits for the public — for instance, projectors. Of their former fleet of 10-12 projectors, they have so far replaced four, though they’ll need to replace a few more soon. Bigger pieces will have to wait, though MakerBot has loaned the non-profit a Replicator.

Although the equipment restoration is happening bit by bit, working with half of what they once had has slowed down the rebuilding of the space; it will probably take until the end of the year or so before the organization is fully back up to speed. Donations are certainly still needed and very much welcomed.

The other challenge Eyebeam is facing is the restoration of their archives. Thanks to the volunteers who came in right after the hurricane, the cleaning process happened right away, which helped enormously. Specialists washed salt and oil and other things that could cause damage from tapes, drives, and discs, and whatever was salvageable has been stabilized. What remains is to — carefully — transfer the information from these older formats to a longer-lasting digital format.

Of the work that was submerged, some of it was on media that was fragile in the best of circumstances, so those are of higher priority to transfer first. With some of the material, like VHS and mini-DV tapes, they won’t know how much damage there has been until transfer is attempted. So the transfer needs to be done professionally and delicately, and that’s a very expensive process.

To help with that process, Eyebeam has applied to the Institute of Museum and Library Services (a government agency) for a grant that would provide monetary support for a strategic plan to move forward. They’ve put together an advisory committee that includes several high-profile experts from institutions such as MoMA, Rhizome, and Cornell’s Media Library. If the funding is granted, the project will start in October.

Archival media conservation is, of course, a challenge faced by many organizations all over the world. Says Jones, “Hopefully, not only will this plan help us, but it will also be something we can pass on to other institutions.”

While no one would say they were glad for Hurricane Sandy, it has clearly forced some important questions about priorities and practices. Eyebeam already has a legacy as a place where advances have been made over the last 15 years. Ensuring that it will be a place where people can access what has been done in digital media during this important period, and into the future, will benefit all of us.

Related:

• After the storm: Putting Eyebeam back together

One of these institutions is the Eyebeam Art + Technology Center in New York, a not-for-profit incubator of new media and design. Each year, Eyebeam hosts two groups of residents for five months each, in addition to several fellows for the full year. Almost 250 artists, designers, and technologists have spent time there since Eyebeam first opened in 1997, many of whom we at O’Reilly have known and admired.

Hurricane Sandy brought more than three feet of water, chemicals, and outside debris sweeping into the streets and buildings on the west side of Manhattan. At Eyebeam, this spelled disaster for much of the equipment and archives on their ground floor. The disaster was compounded by the fact that none of this material was covered by flood insurance.

The main space is currently filled with dried-out computers, projectors, mixers and other audio equipment, and two scissor-lifts used for accessing the upper reaches of their 18-foot-high space. Volunteers are looking over and hand-inspecting it all to figure out what can be salvaged; ultimately, they’ll have to find ways to repair, replace, or do without each item.

As for the archives, volunteers have sorted and washed each piece, and begun the task of cataloging what can be preserved. Alumni will be contacted to see if they can provide copies of their work, but most of the material is now very fragile and will need to be digitized and transferred to more stable formats, a time-consuming and expensive process that is expected to take a year or more.

But with all of this comes the opportunity for Eyebeam to reconsider their goals and how they can make their archives available to a wider audience than before.

Events at Eyebeam will resume this week with an #ArtsTech Meetup on 3D printing; larger public events will resume in December. They’re also planning a special series of events to present material from the archives during the week of January 7. Stay tuned to their events page for forthcoming details on that series.

In the meantime, please consider making a donation. Some of you may have enjoyed Eyebeam’s space when they hosted our Data After Dark event celebrating data visualization as part of the 2011 Strata conference in New York. We at O’Reilly appreciate and support all they add to the tech community, and hope you will too.

I didn’t have to think for long. To me, the thing that seems most like magic isn’t Siri or self-driving cars or augmented reality displays. It’s 3D printing.

My reasons are different than you might think. Yes, it’s amazing that, with very little skill, we can manufacture complex objects in our homes and workshops that are made from things like plastic or wood or chocolate or even titanium. This seems an amazing act of conjuring that, just a short time ago, would have been difficult to imagine outside of the “Star Trek” set.

But the thing that makes 3D printing really special is the magic it allows us to perform: the technology is capable of making us more human.

I recently had the opportunity to lay out this idea in an Ignite talk at Strata Rx, a new conference on data science and health care that I chaired with Colin Hill. Here’s the talk I gave there (don’t worry: like all Ignite talks, it’s only five minutes long).

In addition to the applications mentioned in my talk, there are even more amazing accomplishments just over the horizon. Doctor Anthony Atala, of the Wake Forest University School of Medicine, recently printed a human kidney onstage at TED.

This was not actually a working kidney — one of the challenges to creating working organs is building blood vessels that can provide cells on the inside of the organ structure with nutrients; right now, the cells inside larger structures tend to die rapidly. But researchers at MIT and the University of Pennsylvania are experimenting with printing these vessel networks in sugar. Cells can be grown around the networks, and then the sugar can be dissolved, leaving a void through which blood could flow. As printer resolution improves, these networks can become finer.

And 3D printing becomes even more powerful when combined with other technologies. For example, researchers at the Wake Forest Institute of Regenerative Medicine are using a hybrid 3D printing/electrospinning technique to print replacement cartilage.

As practiced by Bespoke Innovations, the WREX team, and others , 3D printing requires a very advanced and carefully honed skillset; it is not yet within reach of the average DIYer. But what is so amazing — what makes it magic — is that when used in these ways at such a level, the technology disappears. You don’t really see it, not unless you’re looking. What you see is the person it benefits.

Technology that augments us, that makes us more than we are even at our best (such as self-driving cars or sophisticated digital assistants) is a neat party trick, and an homage to our superheros. But those that are superhuman are not like us; they are Other. Every story, from Superman to the X-Men to the Watchmen, includes an element of struggle with what it means to be more than human. In short, it means outsider status.

We are never more acutely aware of our own humanity, and all the frailty that entails, as when we are sick or injured. When we can use technology such as 3D printing to make us more whole, then it makes us more human, not Other. It restores our insider status.

Ask anyone who has lost something truly precious and then found it again. I’m talking on the level of an arm, a leg, a kidney, a jaw. If that doesn’t seem like magic, then I don’t know what does.

As Travis Beacham put it on Twitter:

A jet-fired hover crane just lowered a nuclear robot bigger than my car onto Mars. Then it emailed us pics, from the other side of the sun.

— Travis Beacham (@travisbeacham) August 6, 2012

Although there were tearful hugs and high-fives and all manner of cheering when “Touchdown!” was called, the wonderment built to a real climax when the first thumbnail image came through. It was small, in black and white, and showed the Martian horizon in the background, with the wheel of the rover in the foreground.

Shortly thereafter, a slightly larger version was displayed: still black and white, but with enough resolution to show dust on the glass. A second one followed a few minutes later, showing the rover’s shadow on the ground. Cue the “pics or it didn’t happen” jokes, as well as the rapid proliferation of Photoshopped spoofs.

One of the first images from the Curiosity rover.

In our micro-culture of the moment, obsessed with photo sharing and images, this tiny thumbnail still seemed like a miracle (albeit a required one). A picture really is worth a whole lot of words. But have you ever stopped to think about what it takes to plan for that from Mars?

We take for granted being able to snap a great-looking picture and send it wirelessly to almost anywhere we want with the tap of a few icons, but transmitting images back from another planet is a complicated process.

I couldn’t help but think about the images that came back from the Phoenix lander in 2008, and the excellent chapter J.M. Hughes, principle software engineer for the imaging software on Phoenix, wrote in Beautiful Data:

The challenge was to devise a way to download the image data from each of the cameras, store the data in a pre-allocated memory location, process the data to remove known pixel defects, crop and/or scale the images, perform any commanded compression, and then slice-and-dice it all up into packets for hand-off to the main computer’s downlink manager task for transmission back to Earth.

And all of this must be done carefully, sparingly, in order to conserve resources. As Hughes put it, “A spacecraft is an exercise in applied minimalism: just enough to do the job and no more.”

In honor of the Curiosity’s inspiring success, we’re making Hughes’ chapter available here. Reading about some of the design trade-offs required in building and successfully deploying the imaging software on a Mars spacecraft makes Curiosity’s achievement all the more amazing.

Images from the Curiosity rover can be found here.

Curiosity image: NASA/JPL-Caltech

Beautiful Data — Learn from the best data practitioners in the field just how wide-ranging — and beautiful — working with data can be. Join 39 contributors as they explain how they developed simple and elegant solutions on projects ranging from a Mars lander to a Radiohead video.

Save 50% on the ebook edition with the code PBDMARS.

How are you going to make sense of all that information efficiently so you can make a good decision? Data visualization is an important answer to that question.

However, not all visualizations are actually that helpful. You may be all too familiar with lifeless bar graphs, or line graphs made with software defaults and couched in a slideshow presentation or lengthy document. They can be at best confusing, and at worst misleading. But the good ones are an absolute revelation.

The best data visualizations are ones that expose something new about the underlying patterns and relationships contained within the data. Understanding those relationships — and being able to observe them — is key to good decision making. The Periodic Table is a classic testament to the potential of visualization to reveal hidden relationships in even small datasets. One look at the table, and chemists and middle school students alike grasp the way atoms arrange themselves in groups: alkali metals, noble gasses, halogens.

If visualization done right can reveal so much in even a small dataset like this, imagine what it can reveal within terabytes or petabytes of information.

Types of visualization

It’s important to point out that not all data visualization is created equal. Just as we have paints and pencils and chalk and film to help us capture the world in different ways, with different emphases and for different purposes, there are multiple ways in which to depict the same dataset.

Or, to put it another way, think of visualization as a new set of languages you can use to communicate. Just as French and Russian and Japanese are all ways of encoding ideas so that those ideas can be transported from one person’s mind to another, and decoded again — and just as certain languages are more conducive to certain ideas — so the various kinds of data visualization are a kind of bidirectional encoding that lets ideas and information be transported from the database into your brain.

Explaining and exploring

An important distinction lies between visualization for exploring and visualization for explaining. A third category, visual art, comprises images that encode data but cannot easily be decoded back to the original meaning by a viewer. This kind of visualization can be beautiful, but it is not helpful in making decisions.

Visualization for exploring can be imprecise. It’s useful when you’re not exactly sure what the data has to tell you and you’re trying to get a sense of the relationships and patterns contained within it for the first time. It may take a while to figure out how to approach or clean the data, and which dimensions to include. Therefore, visualization for exploring is best done in such a way that it can be iterated quickly and experimented upon, so that you can find the signal within the noise. Software and automation are your friends here.

Visualization for explaining is best when it is cleanest. Here, the ability to pare down the information to its simplest form — to strip away the noise entirely — will increase the efficiency with which a decision maker can understand it. This is the approach to take once you understand what the data is telling you, and you want to communicate that to someone else. This is the kind of visualization you should be finding in those presentations and sales reports.

Visualization for explaining also includes infographics and other categories of hand-drawn or custom-made images. Automated tools can be used, but one size does not fit all.

Microsoft SQL Server is a comprehensive information platform offering enterprise-ready technologies and tools that help businesses derive maximum value from information at the lowest TCO. SQL Server 2012 launches next year, offering a cloud-ready information platform delivering mission-critical confidence, breakthrough insight, and cloud on your terms; find out more at www.microsoft.com/sql.

Your customers make decisions, too

While data visualization is a powerful tool for helping you and others within your organization make better decisions, it’s important to remember that, in the meantime, your customers are trying to decide between you and your competitors. Many kinds of data visualization, from complex interactive or animated graphs to brightly-colored infographics, can help your customers explore and your customer service folks explain.

That’s why all kinds of companies and organizations, from GE to Trulia to NASA, are beginning to invest significant resources in providing interactive visualizations to their customers and the public. This allows viewers to better understand the company’s business, and interact in a self-directed manner with the company’s expertise.

As big data becomes bigger, and more companies deal with complex datasets with dozens of variables, data visualization will become even more important. So far, the tide of popularity has risen more quickly than the tide of visual literacy, and mediocre efforts abound, in presentations and on the web.

But as visual literacy rises, thanks in no small part to impressive efforts in major media such as The New York Times and The Guardian, data visualization will increasingly become a language your customers and collaborators expect you to speak — and speak well.

Do yourself a favor and hire a designer

It’s well worth investing in a talented in-house designer, or a team of designers. Visualization for explaining works best when someone who understands not only the data itself, but also the principles of design and visual communication, tailors the graph or chart to the message.

Whether it’s text or visuals, important translations require more than basic tools.

To go back to the language analogy: Google Translate is a powerful and useful tool for giving you the general idea of what a foreign text says. But it’s not perfect, and it often lacks nuance. For getting the overall gist of things, it’s great. But I wouldn’t use it to send a letter to a foreign ambassador. For something so sensitive, and where precision counts, it’s worth hiring an experienced human translator.

Since data visualization is like a foreign language, in the same way, hire an experienced designer for important jobs where precision matters. If you’re making the kinds of decisions in which your customer, product, or profit hangs in the balance, you can’t afford to base those decisions on incomplete or misleading representations of the knowledge your company holds.

Your designer is your translator, and one of the most important links you and your customers have to your data.

Related:

• Beautiful Visualization: Looking at Data through the Eyes of Experts (book)
• When judging visualizations, intent matters
• Redesigning the New York City subway map
• Visualization Deconstructed series
• Visualization of the Week series

Tell me a bit about your approach to big data analytics.

Arnab Gupta: Our company is a science-oriented company, and the core belief is that behavior — human or otherwise — can be mathematically expressed. Yes, people make irrational value judgments, but they are driven by common motivation factors, and the math expresses that.

I look at the so-called “big data phenomenon” as the instantiation of human experience. Previously, we could not quantitatively measure human experience, because the data wasn’t being captured. But Twitter recently announced that they now serve 350 billion tweets a day. What we say and what we do has a physical manifestation now. Once there is a physical manifestation of a phenomenon, then it can be mathematically expressed. And if you can express it, then you can shape business ideas around it, whether that’s in government or health care or business.

How do you handle rapidly increasing amounts of data?

Arnab Gupta: It’s an impossible battle when you think about it. The amount of data is going to grow exponentially every day, ever week, every year, so capturing it all can’t be done. In the economic ecosystem there is extraordinary waste. Companies spend vast amounts of money, and the ratio of investment to insight is growing, with much more investment for similar levels of insight. This method just mathematically cannot work.

So, we don’t look for data, we look for signal. What we’ve said is that the shortcut is a priori identifying the signals to know where the fish are swimming, instead of trying to dam the water to find out which fish are in it. We focus on the flow, not a static data capture.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 30% on registration with the code STN11RAD

What role does visualization play in the search for signal?

Arnab Gupta: Visualization is essential. People dumb it down sometimes by calling it “UI” and “dashboards,” and they don’t apply science to the question of how people perceive. We need understanding that feeds into the left brain through the right brain via visual metaphor. At Opera Solutions, we are increasingly trying to figure out the ways in which the mind understands and transforms the visualization of algorithms and data into insights.

If understanding is a priority, then which do you prefer: a black-box model with better predictability, or a transparent model that may be less accurate?

Arnab Gupta: People bifurcate, and think in terms of black-box machines vs. the human mind. But the question is whether you can use machine learning to feed human insight. The power lies in expressing the black box and making it transparent. You do this by stress testing it. For example, if you were looking at a model for mortgage defaults, you would say, “What happens if home prices went down by X percent, or interest rates go up by X percent?” You make your own heuristics, so that when you make a bet you understand exactly how the machine is informing your bet.

Humans can do analysis very well, but the machine does it consistently well; it doesn’t make mistakes. What the machine lacks is the ability to consider orthogonal factors, and the creativity to consider what could be. The human mind fills in those gaps and enhances the power of the machine’s solution.

So you advocate a partnership between the model and the data scientist?

Arnab Gupta: We often create false dichotomies for ourselves, but the truth is it’s never been man vs. machine; it has always been man plus machine. Increasingly, I think it’s an article of faith that the machine beats the human in most large-scale problems, even chess. But though the predictive power of machines may be better on a large-scale basis, if the human mind is trained to use it powerfully, the possibilities are limitless. In the recent Jeopardy showdown with IBM’s Watson, I would have had a three-way competition with Watson, a Jeopardy champion, and a combination of the two. Then you would have seen where the future lies.

Does this mean we need to change our approach to education, and train people to use machines differently?

Arnab Gupta: Absolutely. If you look back in time between now and the 1850s, everything in the world has changed except the classroom. But I think we are dealing with a phase-shift occurring. Like most things, the inertia of power is very hard to shift. Change can take a long time and there will be a lot of debris in the process.

One major hurdle is that the language of machine-plus-human interaction has not yet begun to be developed. It’s partly a silent language, with data visualization as a significant key. The trouble is that language is so powerful that the left brain easily starts dominating, but really almost all of our critical inputs come from non-verbal signals. We have no way of creating a new form of language to describe these things yet. We are at the beginning of trying to develop this.

Another open question is: What’s the skill set and the capabilities necessary for this? At Opera we have focused on the ability to teach machines how to learn. We have 150-160 people working in that area, which is probably the largest private concentration in that area outside IBM and Google. One of the reasons we are hiring all these scientists is to try to innovate at the level of core competencies and the science of comprehension.

The business outcome of that is simply practical. At the end of the day, much of what we do is prosaic; it makes money or it doesn’t make money. It’s a business. But the philosophical fountain from which we drink needs to be a deep one.

Associated photo on home and category pages: prd brain scan by Patrick Denker, on Flickr

Related:

• What is data science?
• Dashboards evolve to meet social and business needs
• Video: Roger Ehrenberg on data analysis and user experience
• Thousands of workers are standing by

Recently, BlueCava announced the formation of an advisory board, which includes executives from Facebook, MasterCard, HP, FirstData, Bill to Mobile, and Merchant Warehouse. I caught up with CEO David L. Norris to discuss device identification, reputation technology, online fraud, and consumer privacy.

Our interview follows.

Tell me a bit more about what BlueCava does and how it works.

David Norris: BlueCava provides a platform that enables businesses to identify devices that are coming to their website. First, we identify the device and then we provide additional information about the device that would be useful to our customers in making decisions about how to interact with that device. One application is finding fraud. Another interesting area is social networking sites: a site may choose not to allow certain users to participate if they have a history of trollish behavior.

As we identify devices, we build information about each device. One of the things we can tell about a device is if it’s a shared computer being used by multiples users. We can also determine the specific level of use — whether it’s a household computer in the kitchen with a handful of users or an Internet cafe computer with hundreds of users.

It sounds like BlueCava is largely used to identify negative behavior. Can the technology also be used to identify devices or users with a positive history?

David Norris: In some ways it’s better to identify a good device rather than the bad ones — it’s much harder to mimic or fake a “clean” machine that has no history. So we’ve taken on the task of identifying a broad set of devices. This year, we’ll identify more than 1 billion devices.

From there, among the partnerships we’ve signed up, we’re going to assign direct financial benefits to those with a positive history, such as discounts and rewards. We’ll be announcing further details soon. For site managers, it you have a historical reputation that’s good, there’s an opportunity to reduce some of the costs associated with interacting with you, like performing extensive background checks. So they can afford to pass some of those savings on to users who merit it.

What about the privacy issues associated with device identification?

David Norris: We do not collect any personal information. We don’t collect Social Security numbers or email addresses. We identify devices and we characterize a device’s behavior. For devices with GPS receivers built in, we collect information at a ZIP-code level, not a granular level. That would be a violation of privacy.

We’ve also implemented what the FTC is calling “do not track,” so users can either opt out or set their preferences when it comes to online marketing.

There’s a difference between being identified and being tracked: if you turn tracking off, we can still identify a device but we don’t keep track of which websites it’s been to.

Since no system is perfect, what are the remedies available to users whose devices or histories are misidentified?

David Norris: If a question comes up about a particular device, the user can go to the merchant or site owner, who can then escalate the issue in a review queue. It becomes a human process at that point.

So BlueCava is not making direct recommendations about user accounts?

David Norris: We’re very careful not to position ourselves as a fraud solution. We are a tech company that can be part of an existing fraud solution, but device identification is only part of the story. We’re gathering information that’s already available and has been used for years by other companies. What we’re doing differently is using it in a unique way.

Imagine that you’re a store owner, and one day someone walks into your store and then walks out. The next day, they walk in again, and your recognize them. You’d do that naturally based on hair color, eye color, the shape of their face, etc. And you could recognize them even if they were wearing a different shirt, because you know that their shirt can change but their face won’t. We do the same thing with devices. Our technology is adaptive, and allows for change to occur. But it’s up to each individual client how to use that information.

Some users may find this kind of device identification intimidating because it seems like “magical spying.” What would you say to them?

David Norris: Cookies used to seem magical too. But then people got used to the idea of them.

Our technology, I believe, will replace cookies eventually. It just observes your machine instead of reaching into it and dropping something there.

Device identification is an improvement over cookies in part because if you choose to opt out, you’re opted out and that’s it. If you opt out using cookies, the system actually drops an opt-out cookie on your machine — if you clear your cookies, then you’re opted back in! Also, you have to opt out on multiple browsers. From a device identification standpoint, it’s much cleaner: you opt out once and it’s done.

This interview was edited and condensed.

Jud will be speaking at Strata next week on a panel titled “What’s Mine is Yours: the Ethics of Big Data Ownership.”

If you’re attending Strata, you can also find out more about growing business of data marketplaces at a “Data Marketplaces” panel with Ian White of Urban Mapping, Peter Marney of Thomson Reuters, Moe Khosravy of Microsoft, and Dennis Yang of Infochimps.

My interview with Jud follows.

Why is social media data important? What can we do with it or learn from it?

Jud Valeski: Social media today is the first time a reasonably large population has communicated digitally in relative public. The ability to programmatically analyze collective conversation has never really existed. Being able to analyze the collective human consciousness has been the dream of researchers and analysts since day one.

The data itself is important because it can be analyzed to assist in disaster detection and relief. It can be analyzed for profit in an industry that has always struggled to pinpoint how and where to spend money. It can be analyzed to determine financial market viability (stock trading, for example). It can be analyzed to understand community sentiment, which has political ramifications; we all want our voices heard in order to shape public policy.

What are some of the most common or surprising queries run through Gnip?

Jud Valeski: We don’t look at the queries our customers use. One pattern we have seen, however, is that there are some people who try to use the software to siphon as much data as possible out of a given publisher. “More data, more data, more data.” We hear that all the time. But how our customers configure the Gnip software is up to them.

Strata: Making Data Work, being held Feb. 1-3, 2011 in Santa Clara, Calif., will focus on the business and practice of data. The conference will provide three days of training, breakout sessions, and plenary discussions — along with an Executive Summit, a Sponsor Pavilion, and other events showcasing the new data ecosystem.

Save 30% off registration with the code STR11RAD

With Gnip, customers can choose the data sources they want not just by site but also by category within the site. Can you tell me more about the options for Twitter, which include Decahose, Halfhose, and Spritzer?

Jud Valeski: We tend to categorize social media sources into three buckets: Volume, Coverage, or Both. Volume streams provide a consumer with a sampled rate of volume (Decahose is 10%, for example, while a full firehose is 100% of some service’s activities). Statisticians and analysts like the Volume stuff.

Coverage streams exist to provide full coverage of a certain set of things (e.g., keywords, or the User Mention Stream for Twitter). Advertisers like Coverage streams because their interests are very targeted. There are some products that fall into both categories, but Volume and Coverage tend to describe the overall view.

For Twitter in particular, we use their algorithm as described on their dev pages, adjusted for each particular volume rate desired.

Gnip is currently the only licensed reseller of the full Twitter firehose. Are there other partnerships coming up?

Jud Valeski: “Currently” is the operative word here. While we’re enjoying the implied exclusivity of the current conditions, we fully expect Twitter to grow its VAR tier to ensure a more competitive marketplace.

From my perspective, Twitter enabling VARs allows them to focus on what is near and dear to their hearts — developer use cases, promoted Tweets, end users, and the display ecosystem — while enabling firms focused on the data-delivery business to distribute underlying data for non-display use. Gnip provides stream enrichments for all of the data that flows through our software. Those enrichments include format and protocol normalization, as well as stream augmentation features such as global URL unwinding. Those value-adds make social media API integration and data leverage much easier than doing a bunch of one-off integrations yourself.

We’re certainly working on other partnerships of this level of significance, but we have nothing to announce at this time.

What do you wish more people understood about data markets and/or the way large datasets can be used?

Jud Valeski: First, data is not free, and there’s always someone out there that wants to buy it. As an end-user, educate yourself with how the content you create using someone else’s service could ultimately be used by the service-provider.

Second, black markets are a real problem, and just because “everyone else is doing it” doesn’t mean it’s okay. As an example, botnet-like distributed IP address polling infrastructure is commonly used to extract more data from a publisher’s service than their API usage terms allow. While perhaps fun to build and run (sometimes), these approaches clearly result in aggregated pools of publisher data that the publisher never intended to promote. Once collected, the aggregated pools of data are sold to data-hungry analytics firms. This results in end-user frustration, in that the content they produced was used in a manner that flagrantly violated the terms under which they signed up. These databases are frequently called out as infringing on privacy.

Everyone loves a good Robin Hood story, and that’s how I’d characterize the overall state of data collection today.

How has real-time data changed the field of customer relationship management (CRM)?

Jud Valeski: CRM firms have a new level of awareness. They no longer rely exclusively on dated user studies. A customer service rep may know about your social life through their dashboard the moment you are connected to them over the phone.

I ultimately see the power of understanding collective consciousness in responding to customer service issues. We haven’t even scratched the surface here. Imagine if Company X reached out to you directly every time you had a problem with their product or service. Proactivity can pay huge dividends. Companies haven’t tapped even 10% of the potential here, and part of that is because they’re not spending enough money in the area yet.

Today, “social” is a checkbox that CRM tools attempt to check off just to keep the boss happy. Tomorrow, social data and metaphors will define the tools outright.

Have you learned anything as a social media user yourself from working on Gnip? Is there anything social media users should be more aware of?

Jud Valeski: Read the terms of service for social media services you’re using before you complain about privacy policies or how and where your data is being used. Unless you are on a private network, your data is treated as public for all to use, see, sell, or buy. Don’t kid yourself. Of course, this brings us all the way back around to black markets. Black markets — and publishers’ generally lackadaisical response to them — cloud these waters.

If you can’t make it to Strata, you can learn more about the architectural challenges of distributing social and location data across the web in real time, and how Gnip has evolved to address those challenges, in Jud’s contribution to “Beautiful Data.”

Infopocalypse?

As a former Bostonian, I well remember the Big Dig: a project to sink the city’s Central Artery underground and add tunnels and a bridge to relieve traffic congestion. The effort consumed (not unpredictably) several years and billions of dollars more than originally projected. And by the time the dust cleared, traffic had increased so much that congestion was just as bad, if not worse, than when the project began.

As with Massachusetts and vehicles, so with government and data. Fittingly, the Boston Phoenix this week published a look at the challenge of government data.

Digital storage is not a natural resource. The amount of information that government agencies may be required to keep — from tweets and emails to tax histories — is growing faster than the capacity for storage.

While the Obama administration has made strides to address this situation by forming the Office of Government Information Services (OGIS) in 2009 and appointing Vivek Kundra as Chief Information Officer, the need is so large as to remain overwhelming at the moment. From the same Phoenix article:

The United States Census Bureau alone maintains about 2560 terabytes of information — more data than is contained in all the academic libraries in America, and the equivalent of about 50 million four-door filing cabinets of text documents. In addition to the federal deluge, tens of thousands of municipal and state facilities maintain data ranging from driver’s-license pics to administrative e-mails — or at least they’re required to.

More and more, huge storage requirements meet staffing cuts and tight budgets in a complicated showdown. Many municipal governments have IT staffs of one or two people, if they have IT staffs at all.

This, of course, leads to a lot of outsourcing and privatization, which come with personnel and expertise benefits, but also security drawbacks. Will government digitization and data transport us into the future, or become another big dig against a rising tide?

Strata: Making Data Work, being held Feb. 1-3, 2011 in Santa Clara, Calif., will focus on the business and practice of data. The conference will provide three days of training, breakout sessions, and plenary discussions — along with an Executive Summit, a Sponsor Pavilion, and other events showcasing the new data ecosystem.

Save 30% off registration with the code STR11RAD

This data was raised in a barn

Of course, privatization may save the day after all, if it can significantly lower the cost of data centers by saving power. That’s one goal of Microsoft’s new data center in Quincy, Wash.: it will use outside air for cooling (known as “air-side economizing”), and house the racks in a barn-like building that protects the servers from wind and rain but is otherwise “virtually transparent to ambient outdoor conditions.”

These new server farms will also make use of Microsoft’s IT Pre-Assembled Components (ITPACs), which allow for flexibility and scaling, and will help keep costs down even further.

Microsoft, like Intel in New Mexico and Red Rocks Data Center and Sun Microsystems in Colorado, began experimenting with air-side economizing in late 2008 and 2009.

Intel went on to publish a white paper on air economization as well as a Proof of Concept video in which they report lowering power costs by nearly 74 percent. Red Rocks Data has since closed, but first reported that during the coolest months of the year, “our savings are averaging about $1,600 a month on a $5,000 total bill.”

With or without a tractor shed, many more of these air-cooled data centers with a modular approach are likely to be built in the coming years. Microsoft expects to open others in Virginia and Iowa in 2011, and they likely will not be alone.

Maybe the White House should build itself a barn.

Resource room

If you’re headed to Strata in a couple of weeks and find yourself in need of some anticipatory reading for the flight, download the recent Big Data reports from PricewaterhouseCoopers and NESTA (the National Endowment for Science, Technology and the Arts in the UK).

The NESTA report lays out some of the key concepts and threads from its November 2010 event, “The Power and Possibilities of Big Data.” You can also watch video from the event, which brought together folks like Hans Peter Brøndmo from Nokia, Haakon Overli from Dawn Capital, Max Jolly from dunnhumby, and Megan Smith from Google.

The PwC issue is aimed at CIOs and covers “the techniques behind low-cost distributed computing that have led companies to explore more of their data in new ways.” Several of these articles will be great background before heading off to Strata — hope to see you there!

Does this place accept returns?

On Monday, Stack Overflow announced that they have moved the Stack Exchange Data Explorer (SEDE) off of the Windows Azure platform and onto in-house hardware.

SEDE is an open source, web-based tool for querying the monthly data dump of Creative Commons data from its four main Q&A sites (Stack Overflow, Server Fault, Super User, and Meta) as well as other sites in the Stack Exchange family. The primary reason given (within a polite write-up by Jeff Atwood and SEDE lead Sam Saffron), was the desire to have fine-tuned control over the platform.

When you are using a [Platform-as-a-Service] you are giving up a lot of control to the service provider. The service provider chooses which applications you can run and imposes a series of restrictions. … It was disorienting moving to a platform where we had no idea what kind of hardware was running our app. Giving up control of basic tools and processes we use to tune our environment was extremely painful.

While the support that comes with Platform-as-a-Service was acknowledged, it seems that the ability to better automate, adjust, and perpetuate processes and systems with more fine-grained control won out as a bigger convenience.

Where did you get that lovely platform?

Of course, one company’s headache is another’s dream. Netflix, a company known for playing with big data and crowdsourcing solutions “before it was cool,” posted on Tuesday the four reasons they’ve chosen to use Amazon Web Services (AWS) as their platform and have moved onto it over the last year.

Laudably, the company states that it viewed its tremendous recent growth (in terms of both members and streaming devices) as a license to question everything in the necessary process of re-architecting. Instead of building out their own data centers, etc., they decided to answer that set of questions by paying someone else to worry about it.

Also to their credit, Netflix has enough self-awareness to know what they are and aren’t good at. Building top-notch recommendation systems and providing entertainment? You betcha. Predicting customer growth and device engagement? Not so much.

How many subscribers would you guess used our Wii application the week it launched? How many would you guess will use it next month? We have to ask ourselves these questions for each device we launch because our software systems need to scale to the size of the business, every time.

Self-awareness is in fact the primary lesson in both Netflix’s and Stack Exchange’s platform decisions. If you feel your attention is better spent elsewhere, write a check. If you’ve got the time and expertise to hone your hardware, roll your own.

[Of course, Netflix doesn't go for the pre-packaged solutions every time. They also posted recently about why they love open source software, and listed among the projects they make use of and contribute back to: Hadoop, Hive, HBase, Honu, Ant, Tomcat, Hudson, Ivy, Cassandra, etc.]

With what shall we shop?

The New York Times this week released a cool group of interactive maps based on data collected in the Census Bureau’s American Community Survey (ACS) from 2005 to 2009. Data is compared against the 2000 census to uncover rates of change.

[While similar to the census, the ACS is conducted every year instead of every 10 years. The ACS includes only a sampling of addresses instead of a comprehensive inventory. It covers much of the same ground on population (age, race, disability status, family relationships), but it also asks for information that is used to help make funding distribution decisions about community services and institutions.]

The Times maps explore education levels; rent, mortgage rates, and home values; household income; and racial distribution. Viewers can select among 22 maps in these four categories, and then pan and zoom to view national, state, or local trends down to the level of individual census tracts.

Above is the national view of the map that looks at change in median household income. The ACS website itself provides some maps displaying the survey numbers from the 2000 census and the 2005-2009 survey, as well as a listing of data tables.

The Times map shows the uneven way in which these numbers have gone up or down in various parts of the country, with some surprising results that are worth exploring. Note that the blue regions are places where income has dropped, and the yellow regions are places where it has increased. (No wonder a lot of us are getting creative with holiday shopping.)

If this kind of research floats your boat, check out Social Explorer, the mapping tool used to create the New York Times maps.

Even markets like to buy things

The emerging landscape of custom data markets is already shifting as Infochimps recently announced the acquisition of Data Marketplace, a start-up incubated at Y Combinator.

While Stewart Brand may be right in thinking information wants to be free, there’s also enormous value to be added by aggregating, structuring, and packaging data, as well as in matching up buyers with sellers. That’s the main service Data Marketplace aims to provide, particularly in the field of financial data.

At Infochimps, information is offered a la carte, and many of the site’s datasets are offered for free. These include sets as diverse as “Word List – 100,000+ official crossword words (Excel readable)“, “Measuring Worth: Interest Rates – US & UK 1790-2000“, and “Retrosheet: Game Logs (play-by-play) for Major League Baseball Games.” Data Marketplace is a bit different, in that it allows users to enter requests for data (with a deadline and budget, if desired) and then matches up would-be buyers with data providers.

Infochimps has said that Data Marketplace, which is less than a year old, will continue to operate as a standalone site, although its founders Steve DeWald and Matt Hodan will depart for new projects.

If you’re interested in the burgeoning business of aggregated datasets, be sure to check out the Data Marketplaces panel I’ll be moderating at Strata in February.

Not yet signed up for Strata? Register now and save 30% with the code STR11RAD.