Ipeirotis’ findings are surprising: consumers will pay more for the same product if the seller’s reviews are good, certain types of negative reviews actually boost sales, and spelling plays an important role.

Our interview follows.

How important are product reviews on Amazon? Can they give sellers more pricing power?

Panagiotis Ipeirotis: The reviews have a significant effect. When buying online, customers are not only purchasing the product, they’re also inherently buying the guarantee of a seamless transaction. Customers read the feedback left from other buyers to evaluate the reputation of the seller. Since customers are willing to pay more to buy from merchants with a better reputation — something we call the “reputation premium” — that feedback tends to have an effect on future prices that the merchant can charge.

What are some of the most influential phrases?

Panagiotis Ipeirotis: “Never received” is a killer phrase in terms of reputation. It reduced the price a seller can charge by an average of $7.46 in the products examined. “Wonderful experience” is one of the most positive, increasing the price a seller can charge by $5.86 for the researched products.

How can very positive reviews be bad for sales?

Panagiotis Ipeirotis: Extremely positive reviews that contain no concrete details tend to be perceived as non-objective — written by fanboys or spammers. We observed this mainly in the context of product reviews, where superlative phrases like “Best camera!” with no further details are actually seen negatively.

Can a negative review ever be good for sales?

Panagiotis Ipeirotis: It can when the review is overly negative or criticizes aspects of the product that are not its primary purpose — the video quality in an SLR camera, for example. Or, when customers have unreasonable expectations: “Battery life lasts only for two days of shooting.” Readers interpret these types of negative comments as “This is good enough for me,” and it decreases their uncertainty about the product.

What is the effect of badly written reviews on sales?

Panagiotis Ipeirotis: Reviews containing spelling and grammatical errors consistently result in suboptimal outcomes, like lower sales or lower response rates. That was a fascinating but, in retrospect, expected finding. This holds true in a wide variety of settings, from reviews of electronics to hotels. It’s even the case when examining email correspondence about a decision, such as whether or not to hire a contractor.

We don’t know the exact reason yet, but the effect is very systematic. There are several possible explanations:

• Readers think that the customers who buy this product are uneducated, so they don’t buy it.
• Reviews that are badly written are considered unreliable and therefore increase the uncertainty about the product.
• Badly written reviews are unsuccessful attempts to spam and are a signal that even the other good reviews may not be authentic.

What’s the relationship between the product attributes discussed in reviews and the attributes that lead to sales?

Panagiotis Ipeirotis: We observed that the aspects of a product that drive the online discussion are not necessarily the ones that define consumer decisions to buy it. For example, “zoom” tends to be discussed a lot for small point-and-shoot cameras. However, very few people are influenced by the zoom capabilities when it comes down to deciding which camera to buy.

This interview was edited and condensed.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

Related:

• “Big Data, Stupid Decisions: The Importance Of Measuring The Right Thing” (Panagiotis Ipeirotis’ Strata Jumpstart 2011 presentation)
• When good feedback leaves a bad impression
• Visualization of the Week: Amazon book recommendations

Malicious types of software, like viruses, worms, and trojans, are the main tools used to harvest personal data. Cyber criminals also use social engineering techniques, such as phishing emails populated with data gleaned from social networks, to trick people into providing further details. In the interview below, Goodman outlines some of the other ways organized criminals and terrorists are harnessing data for nefarious ends.

What motivates data criminals?

Marc Goodman: Anything that would motivate someone to join a startup would motivate a criminal. They want money, shares in the business, a challenge. They don’t want a 9-to-5 environment. They also want the respect of their peers. They have an us-against-them attitude; they’re highly innovative and adaptive, and they never take the head-on approach. They always find clever and imaginative ways to go about something that a good person would never have considered.

What type of personal data is most valuable to criminals?

Marc Goodman: The best value is a bank account takeover. A standard credit card might cost a criminal only $10, but for $700 they could buy details of a bank account with $50,000 in it, money that could be stolen in just one transaction.

European credit cards tend to cost more than American credit cards since Europeans are much better at guarding their data and have legislation in place, such as the European directive on data privacy, which prohibits the aggregation and long-term storage of personally identifiable information.There’s also a universal identifier for Americans — the social security number — but the same thing doesn’t exist from a pan-European perspective.

How is data crime more scalable than traditional crime?

Marc Goodman: Data crime can be scripted and automated. If you were to take a gun or a knife and stand on a street corner, there are only so many people you can rob. You have to do the crime, run away from the scene, worry about the police, etc. You can’t walk into Wembley Stadium with a gun and say, “Everybody, put your hands up,” but you can do the equivalent from a cyber-crime perspective.

One of the reasons why cyber crime thrives is that it’s totally international whereas law enforcement is totally national. Now, the person attacking you can be sitting in New York or Tokyo or Botswana. The ability to conduct business without getting on a plane is an awesome advantage for international organized crime.

Strata 2012 — The 2012 Strata Conference, being held Feb. 28-March 1 in Santa Clara, Calif., will offer three full days of hands-on data training and information-rich sessions. Strata brings together the people, tools, and technologies you need to make data work.

Save 20% on registration with the code RADAR20

How has cyber crime evolved?

Marc Goodman: In the 1970s, you had to be a clever hacker and create your own scripts. Now all of that stuff can be bought off the shelf. You can buy a package of crimeware and put in the email addresses or the domain that you want to attack via a nice user interface. It’s really plug-and-play criminality.

You claim that the 2008 Mumbai attackers used real-time data gathering from social networks and other media. How do terrorists use data?

Marc Goodman: Since the Internet arrived, terrorists have been advertising, doing PR, recruiting, and fundraising, all online. But this was the first time that we had seen terrorists use technology to the full extent that this group did during the incident. They had mobile phones and satellite phones. The terrorist war room they set up to monitor the media and feed back information in real time to the attackers was a really significant innovation.

They re-engineered the attack mid-incident to kill more people. They were constantly looking for new hostages. Organizations like the BBC and CNN were tweeting to ask people on the ground in Mumbai to contact a producer. People trapped in hotels called the TV stations. All of that information was being tracked by the terrorist war room. There was an Indian minister who was doing a live interview on the Indian Broadcast Network (IBN) while hiding in the kitchen of the ballroom of the Taj Mahal hotel. The war room picked this up and directed the attackers to that part of the hotel where they could find the minister.

What can be done to combat cyber crime?

Marc Goodman: The terrorism problem is very different from the cyber crime problem. Most acts of terrorism are carried out in the real world whereas cybercrime offenses take place in virtual spaces.Governments are pretty good at tracking the terrorists in their own countries, and there is decent international cooperation on terrorism.

What is making things more difficult for governments is that, in the old days, if you tapped somebody’s home phone, you had a good picture of what was going on. Now you don’t know where to look. Are they communicating on Facebook, on Twitter, or having a meeting in World of Warcraft?

Law enforcement needs to develop better systems to deal with the caophony of social media usage during a terrorist attack. The public is getting involved in ways that are, frankly, unhealthy. There was a hostage situation in the U.S. a couple of months ago where a man took a hostage and was sexually assaulting her. He had her trapped in a hotel room with guns and was posting live on Facebook and Twitter. Then the public started to interact with the hostage-taker, tweeting things like, “You wouldn’t kill her. You are not brave enough to do it.” In the past, police could close off several city blocks, put up yellow crime scene tape, close the airspace over the scene, and bring in a trained negotiator. How does law enforcement intervene when there can be a completely disintermediated conversation between the criminal or terrorist and the general public?

Marc Goodman discussed the business of illegal data at Strata New York 2011. His full presentation is available in the following video:

This interview was edited and condensed.

Related:

• From crowdsourcing to crime-sourcing: The rise of distributed criminality
• Why cloud services are a tempting target for attackers
  
• Trend to watch: Formal relationships between governments and hackers
• Cyber warfare: don’t inflate it, don’t underestimate it

EMeter provides metering data management products to utilities. I talked to Aaron DeYonker, global head of product management at eMeter, about how data will transform the way utilities do business. Our interview follows.

How much new data do utilities have to process because of smart metering?

Aaron DeYonker: Currently, most utilities do a standard meter read once a month. With smart meters, utilities have to process data at 15-minute intervals. This is about a 3,000-fold increase in daily data processing for a utility, and it’s just the first wave of the data deluge. The second wave will include granular data from smart appliances, electric vehicles and other metering points throughout the grid. That will exponentially increase the amount of data being generated.

What processes do utilities have to change to deal with the data?

Aaron DeYonker: Utilities tend to be risk adverse and as a result may not have utilized technology for internal business process as much as other industries like telco or banking. Coupled with growing energy demands, utilities need to change a number of business processes beyond billing to handle the amount and type of data they now manage. Outage management, customer service, load research and wholesale electricity market transactions — where utilities buy and sell electricity to each other — are just a few of the impacted areas.

In the case of power outage management, for example, the traditional business process relied on processing customer calls to detect outage hotspots. Now, smart meters can automatically report that they have lost power. That enables more proactive dispatching of resources to fix problems. However, the data needs to be collected, validated and processed quickly to make resolution times as fast as possible. So utilities now need to develop new business processes to enable this.

Strata Conference New York 2011, being held Sept. 22-23, covers the latest and best tools and technologies for data science — from gathering, cleaning, analyzing, and storing data to communicating data intelligence effectively.

Save 30% on registration with the code STN11RAD

Beyond billing, how can utilities put metering data to use?

Aaron DeYonker: The list is endless, but here are a few use cases: customer service, outage management, conservation and efficiency programs, demand response (reducing consumption at peak demand times), demand forecasting, energy “theft” detection, “line-loss” detection, new product offerings (including new rates for electric vehicles and renewables), and peak-load reduction programs.

Let’s take customer service as a specific example. Many utility call centers deal with routine complaints about high bills. Customers can now get notifications that a spike in usage has been detected and the data might even be able to pinpoint the actual culprit within the home, such as an aggressively programmed air conditioner. Alerting the customer before the bill arrives will reduce call volumes and increase customer satisfaction.

What actionable metrics or alerts can you derive from meter data?

Aaron DeYonker: Having access to more detailed data opens the door to setting more accurate metrics, both for the utility and the consumer. With the right data, utilities can better diagnose outages as close to real time as possible and allocate resources as quickly and effectively as possible to resolve problems.The meters will send these events up and the system needs to perform powerful and fast analysis to distinguish true problems from false alarms.

On the business side, many utilities look for ways to reduce the peak demand for energy, which typically occurs in the afternoon and evening during the summer. Programs that have cheaper night-time rates are now possible because metering data is gathered at regular intervals. Measuring the effect of these rates on customer behavior is critical in lobbying regulatory bodies to approve these programs for broader roll out.

In addition, consumers are empowered to regulate their own energy use. That helps reduce unnecessary energy usage and generation. In some jurisdictions, the metering data will in fact eliminate the need for new nuclear power plants based on the expected efficiencies derived from smarter pricing.

How do you see utilities using metering data in the future?

Aaron DeYonker: At the very least, flattening out the daily demand curve will have incalculable benefits on our overall energy infrastructure as demand increases. The use of metering data enables a limitless potential for new products and services for utilities and their customers. Armed with a rich dataset on par with that of the financial services and telco industries, utility companies can model, forecast and prototype with much more power and precision. It’s impossible to envision the extent to which this data is applied to future innovations, but it will be extensive and world changing.

This interview was edited and condensed.

Photo: Seoul by night by Koshyk, on Flickr

Related:

• The super grid. Coming soon to a power outlet near you
• Plugging water leaks with data
• Algorithms are the new medical test
• Trading on sentiment
• With M2M, the machines do all the talking

Beyond clever headlines, OkCupid also uses an unusual pedigree to separate itself from the dating site pack: The business was founded by four Harvard-educated mathematicians.

“It probably scared people when they first heard that four math majors were starting a dating site,” said CEO Sam Yagan during a recent interview. But the founders’ backgrounds greatly influenced how they approached the problem of dating.

“A lot of other dating sites are based on psychology,” Yagan said. “The fundamental premise of a site like eHarmony is that they know the answer. Our approach to dating isn’t that there’s some psychological theory that will be the answer to all your problems. We think that dating is a problem to be solved using data and analytics. There is no magic formula that can help everyone to find love. Instead, we bring value by building a decent-sized platform that allows people to provide information that helps us to customize a match algorithm to each person’s needs.”

OkCupid works by having users state basic preferences and answering questions like “Is it wrong to spank a child who’s been bad?” Users are matched based on the overlap of their answers and how important each question is to both users.

Yagan said data was built into the business model from the beginning. “We knew from the time we started the company that the data we were generating would have three purposes: helping us match people up, attracting advertisers since that was the core of our revenue model, and that the data would also be interesting socially.”

In 2007, the company hired a PR firm to publicize some of its findings, such as the fact that when gas prices rise, users narrow the search radius for matches. “We called dozens of reporters and nobody cared,” Yagan said. So OkCupid fired the PR firm and started publishing their findings on the OkTrends blog. The blog has thus far doubled traffic to the site.

“The blog is partly an advice column, but instead of being written by a psychologist, the data writes itself,” Yagan said. “For example, we don’t tell you that you should or should not use a flash for your profile photo. We just tell you that if you use a flash you’ll look seven years older.”

Web 2.0 Summit, being held October 17-19 in San Francisco, will examine “The Data Frame” — focusing on the impact of data in today’s networked economy.

Save $300 on registration with the code RADAR

I asked Yagan about the data on which OkTrends draws. “We have people’s registration data,” he said. “Then we have stated preferences; the answers that people give to the questions we ask them. We use that kind of data occasionally, but it’s not the core difference that we have. The core difference is in the category of revealed preferences. Imagine if you had a video camera in every bar and you could observe every interaction between two people and see the success rate of that interaction. We essentially have that video camera on our site.”

The reason revealed preferences are so important is that they track real-world behavior — what people really want rather than what they say they want. “When you get 12 messages and you only reply to three of them, you are voting with your time,” Yagan said. “Or when a guy is shorter than you, you don’t reply.”

Mobile adds a new revealed preferences dimension for OkCupid. “As our product gets more mobile and location-aware, we are more likely to be on that date with them,” Yagan said. “Then we can model the kinds of conversations on the site that lead to an in-person meeting.” OkCupid can currently track the five million messages sent every week on the site as well as other revealed preferences, like ratings of profiles.

According to Yagan, OkCupid doesn’t use sophisticated data mining or analytics tools: “Most of it can be done by querying the database and crunching numbers in Excel. The fact that we have four math majors and a full-time statistician means that we take that number crunching very seriously.”

Related:

• Personal data is the future, but does anybody care?
• Want to know where to build a new store? Check your human density data
• Scraping, cleaning, and selling big data
• A new twist on “data-driven site”

Predictive Medical Systems is based in the University of Utah’s medical accelerator, which is attached to a hospital. The system will soon be tested on a trial basis with real patients and ICU physicians.

I recently talked to CEO Bryan Hughes about using data in diagnosis. Our interview follows

What kinds of data is already available from hospital electronic medical records (EMR) and patient monitoring systems?

Bryan Hughes: We require that a hospital be at a certain technological level, in particular that the hospital has an EMR solution that is at minimum classified as Stage 4, or a Computerized Physician Order Entry system. Only about 100 hospitals in the U.S. are at this stage right now.

Once a hospital has achieved this stage, we can integrate with their computer systems and extract the raw data coming from the monitors, lab reports and even nursing notes. We can then perform realtime patient data mining and data analytics.

Our system works behind the scenes constantly analyzing the raw patient data coming in from a variety of sources like chemistry panels, urinalysis, micro biology, respiratory and bedside monitors. We attempt to alert the doctor early of an adverse event such as cardiac arrest, or that a patient might be trending toward an arrhythmia or pneumonia.

Health IT at OSCON 2011 — The conjunction of open source and open data with health technology promises to improve creaking infrastructure and give greater control and engagement for patients. These topics will be explored in the healthcare track at OSCON (July 25-29 in Portland, Ore.)

Save 20% on registration with the code OS11RAD

How does the system integrate into an ICU doctor’s existing routine?

Bryan Hughes: Depending on the technological development of a hospital, doctors either do their rounds in the ICU using a piece of paper or using a bedside computer terminal. Older systems might employ a COW (Computer On Wheels).

For hospitals that are still paper based, they have to first get to the EMR stage.
It is surprising that health care, the largest and quintessential information-based industry, has failed to harness modern information exchange for so long. The oral tradition and handwritten manuscripts remain prevalent throughout most of the sector.

For hospitals that have an EMR, there are still several fundamental problems. The single most daunting problem facing modern doctors is the overwhelming amount of data. Unfortunately, especially with the growing adoption of electronic medical records, this information is disparate and not immediately available. The ability for a clinician to practice medicine is rooted in the ability to make sound decisions on reliable information.

Disparate information makes it hard to connect the dots. Massive amounts of disparate information turns the dots into a confusing sea of blobs. The dots must be connected in a manner that allows the doctor to make immediate and intelligent decisions.

We look at the current trends and progressions of disease states in the now, and then look at what may be happening in the next 24 hours. We then push this information to a mobile device such as an iPad allowing the doctor to see the clinically relevant dots, allowing them to make better decisions in a timely manner.

Eventually we hope to expand to the entire hospital. But for now, the ICU is a big enough problem and a great starting point.

How do you use data to predict outcomes like cardiac arrest?

<p.

Bryan Hughes: We have two first-generation models: cardiac arrest and respiratory failure. We plan to apply our novel techniques to modeling sepsis, renal failure and re-intubation risk.

Without giving away too much of our secret sauce, we use non-hypothesis machine learning techniques, which have proven very promising so far. This approach allows us to eliminate any human “expert” bias from the models. The key then is to ensure that the data we use for development and training is clean. It is only now that medical data is in electronic and structured form that this is becoming readily available.

What kinds of data mining techniques do you use in the product?

Bryan Hughes: We use a variety of techniques. Again, without giving too much away, our approach is to use transparent algorithms rather than a black box approach. We have a patent strategy that allows us to effectively place a white fence around our technology while allowing the academic and medical community to review our results.

How do you judge the accuracy of the algorithms?

Bryan Hughes: To date, our results have been proven using retrospective models (historical ICU monitoring data and outcomes). Our next step is to deploy our technology into a validation trial — a validation trial produces evidence that a test or treatment produces a clinical benefit. That trial is about to start at the University of Utah Medical Center in Salt Lake City.

Once the integration is completed in the next several weeks, we will be running a double-blind, prospective study with patient data. While this is only a validation trial, we are following the FDA guidance. Once the trial is up and running, we plan on expanding the validation trial to include several more hospitals. It will be at least 12 months before we start any formal FDA trial.

How is the system updated over time?

Bryan Hughes; We have developed a unique architecture that allows the system to reduce the experiment to validation cycle to 8 to 10 months. Typically in the medical community, a hypothesis is developed, a model is built and then tested and if valid, a paper is published for peer review. Once the model is accepted, it can have a life span of several years of adoption and application, which is bad because as we know, information and knowledge changes as we learn and understand more. Models need to be consistently re-evaluated and re-examined.

Are any similar systems available?

Bryan Hughes: None in the ICU, or even dealing with patient care, that we have found to date. In other industries, predictive analysis and modeling are pretty common place. Even your spam filter employs many of the techniques that the most sophisticated risk analysis system might use.

Photo: ekg by krzakptak, on Flickr

Related:

• A new challenge looks for a smarter algorithm to improve healthcare
• 3 ways the Internet is shaping healthcare
• Watson’s marketable skills
• Open health data: Spurring better decisions and new businesses

In M2M, devices and sensors communicate with each other or a central server rather than with human beings. These devices often use an embedded SIM card for communication over the mobile network. Applications include automotive, smartgrid, healthcare and environmental usages.

M2M traffic differs from human-generated voice and data traffic. Mobile carriers are adapting by creating entirely new companies for M2M, such as Telenor’s M2M carrier Telenor Connexion, and m2o city, Orange’s joint venture with water giant Veolia. I talked to Göran Brandt, head of business development at Telenor Connexion and Rodolphe Fruges, VP of M2M at Orange Business Services about the future of mobile and M2M.

Why did Telenor start Telenor Connexion?

Göran Brandt: Telenor Connexion was founded in 2008. We knew from our experience with running business-critical applications on the normal mobile infrastructure that it was not good enough. A system originally built to serve voice services, mobile office applications, etc. is not ideal for M2M. This could lead to disturbances or downtime due to normal mobile service windows. For example at night, voice customers are expected to be sleeping.

Why did Orange launch a mobile service operator specifically for water metering data?

Rodolphe Fruges: Smart metering for utilities — water, electricity and gas — is a relatively new market where we see key advances in M2M taking place. To address this market, Orange has joined forces with Veolia Eau, a market leader in the water industry, to create m2o city, a joint venture dedicated to smart metering.

To be clear, m2o city is not a “mobile operator.” That would require a GSM
license. Rather, it is a “service operator” that provides the low-energy radio
network that carries water metering data on behalf of local water distribution
companies.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

How is the data carried in M2M applications different from human-generated data?

Göran Brandt: In M2M applications the amount of data sent and received is normally small. A meter reading equals only a few hundred bytes of data. M2M devices can cause problems if working incorrectly. If hundreds of thousands or even millions of electricity meters act at exactly the same time (they normally have a very precise built-in clock), that would result in network congestion.

Rodolphe Fruges: The difference is not the data itself but where the data originates, in this case self-contained mobile devices. These devices are generating a huge amount of data with more frequency. With m2o city, utility companies are dealing with 700 times more data than before. This is why service providers like Orange, who have the expertise and technical infrastructure to accommodate these data loads, are vital for these companies.

The data also varies between applications. For water metering, we are typically dealing with a very small data set, a few times a day, at regular hours. For security applications, the device can be silent for months before sending a large data payload, in this case video.

What network management strategies, technologies or processes does M2M rely on?

Göran Brandt: For a normal mobile carrier the customer interface is usually the customer help desk. In M2M you need to let your customers into your technical systems, so they can, in real time, see the status of their SIM card population and answer questions like “What country is a specific SIM card in?” and “Is the SIM card connected to a mobile data network or not?”

Rodolphe Fruges: Creating processes and mechanisms are really essential for smart metering. You have to manage millions of devices with very specialized SIM cards across varying environmental conditions. A phone user can easily call a help desk to report a problem with his or her phone, but the device itself cannot make this call.
The challenge on our end is to create automatic mechanisms that can validate
whether a device is working or identify the source of any potential problems.

Certain M2M applications, such as streaming security videos, generate a high volume of data comparable to the data streaming occurring on mobile phones, but you can also find M2M alert applications running SMS data levels. What changes is the number of devices that are being managed rather than the volume of data.

A key issue with smart metering is meters situated in hard-to-reach areas, such as basements where it may be rough to get a strong mobile signal. The
solution is network intermarries, which is like a meshed radio network that grabs the
data from a meter and sends it to a data concentrator. This is exactly the
type of network technology you will see deployed with m2o city. It’s also applicable in other M2M scenarios, like a connected automobile that is roaming across networks. We constantly adapt to the least cost network when roaming or alert our customer when an event, like a vehicle that might be stolen crossing a border, are triggered on the network.

What are the biggest current M2M applications? What do you see developing in the near future?

Göran Brandt: Automotive (cars and trucks), energy (smart metering) and security (burglar alarms).The automotive industry is a sector where Telenor Connexion has extensive experience. We are working in close collaboration with both car manufacturers and telematics service providers to enable cost-efficient and reliable connectivity solutions to vehicles around the world.

There are massive smart metering deployments currently being planned. The energy sector is moving toward renewable energy sources and the implementation of smart grids. Intelligent meters are going to form the foundation of tomorrow’s smart grid infrastructure.

Healthcare monitoring via wireless networks is an emerging application area with huge potential. Tens of millions of patients in Europe alone could potentially benefit from some form of home healthcare monitoring solution, if it had been available. Examples of conditions suitable for remote monitoring are cardiac arrhythmia, hypertension, sleep apnea and diabetes.

Are there obstacles preventing M2M applications from become mainstream?

Göran Brandt: Most M2M customers work in a multi-country or even global scenario (basically all countries where they sell their products), and they expect a single solution covering multiple networks in multiple countries. It’s a challenge to provide flat pricing in multi-country roaming situations. It’s equally challenging to offer M2M customers Service Level Agreements stating exactly what uptime and availability to expect, including roaming networks.

Rodolphe Fruges: The M2M applications are enormous, but it has led to some fragmentation in the market. Verticals such as the automotive industry have been hampered by the number of competitive players trying to outdo one another. A lack of
standardization has also been a deterrent impeding newer and more reliable
M2M solutions.

Photo: Screenshot from Telenor Connexion website

Related:

• Plugging water leaks with data
• The quiet rise of machine learning

Case in point: The financial trading dashboard managed by
Thomson Reuters uses sentiment analysis data from Lexalytics to track news on
20,000 stocks and thousands of commodities. The Lexalytics system
parses text from multiple sources, looking for keywords, tone,
relevance and freshness. The resulting textual analysis (the meaning of the text) and sentiment analysis (the emotions in the text) is then
incorporated into widely used algorithmic trading systems.

Mark Thompson, CEO of McKinley Software (the parent company of Lexalytics), told me more about this emotion-to-data conversion. “Our financial engine is something we developed over an 8-year period, and the main partner for that is Thomson Reuters,” Thompson said. “The Thomson Reuters news passes through our black box and we kick out scores based on 80 different variables for all of the articles.”

Algorithmic trading is automated trading where trading software takes various inputs, or “trading signals,” and uses them to decide what trades to make. Trades are executed in a matter of milliseconds and there is no human intervention. In 2009, algorithmic trading accounted for more than 25 percent of all shares traded on the buy side. No human being can read the latest financial news fast enough to contribute to those buy or sell decisions. That’s where sentiment analysis comes in.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

“By scoring the news, within milliseconds we get a very accurate view of what’s being said about a particular stock or sector,” Thompson said. “Thomson Reuters sells that output to trading houses who then plug this data into their algorithmic trading
models. We have found that we can predict stock market movements. We provide an extra layer of richness that trading staff haven’t been able to get their hands on. Otherwise, you are just doing very two-dimensional quant
processing.”

Rochester Cahan, VP of Global Equity Quantitative Strategy at Deutsche Bank, has been experimenting with the Thomson Reuters system. Cahan told me that he has seen significant improvements in trading performance when the text analysis and sentiment scores are used as trading inputs. In addition, the scores are uncorrelated with existing trading signals — in other words, they provide new information to the trading system.

The most positive sentiment levels (e.g. Apple releases the iPad to universal acclaim) are not necessarily the most useful for trading. The stock price reacts very quickly so it’s difficult to take advantage of the information. However, Cahan said stocks with moderate positivity tend to be overlooked by the market and can make for good buys.

I asked Thompson about the limitations of the sentiment analysis technology. He explained that even human beings don’t agree on the sentiment of an article more than about 85% of the time. “The problem with our kind of engine is trying to get
above 85% accuracy,” Thompson said. “Beyond that level, you get a diminishing return and you need more human intervention. This leaves the human analyst to
pass different types of judgements.”

The competitive edge may be lost if all trading systems use sentiment analysis, but Thompson thinks there is some distance to go before we get to that point. “Everyone has a slightly different way of composing the model and using the news, and there are always advances in the technology,” he said. “But there will come a point when sentiment becomes an ordinary part of the trading mix.”

Photo: ABOVE by Lyfetime, on Flickr

Related:

• With sentiment analysis, context always matters
• Social data is an oracle waiting for a question
• Video: Matthew Russell on social data tools and the promise of the semantic web

What water companies need is an efficient system that can collect and parse monitoring data so leaks can be found and repaired.

Israeli startup TaKaDu aims to fill that niche with its water infrastructure monitoring service. UK-based Thames Water, for example, uses TaKaDu to detect leaks up to 9 days earlier than with its previous system.

In the following interview, TaKaDu’s VP of Marketing Guy Horowitz talks about how monitoring systems use data to plug these costly leaks.

How much revenue is lost to leaks?

Guy Horowitz: There are direct losses and indirect losses. Direct losses are the cost of water and energy. In many countries, the water that is lost — desalinated water for example — is very expensive to produce. Even in countries with abundant water, water has to be pumped, moved and stored, which contributes to the cost of water. A city such as London loses 30% of its water; over 600 million cubic meters. A cubic meter can cost anywhere from a $0.20 to a few dollars, so we’re talking about hundreds of thousands of dollars, if not millions, per day. Add to that the energy required to replace the water lost, and you get a significant pain point.

Indirect losses are often higher than direct losses. Road and property damage, traffic interruptions, paying detection crews and repair crews, and in many cases regulatory fines, amount to hundreds of millions per year in large cities. Detection alone is a mega-expenditure.

Additional loss of revenue is associated with sub-optimized maintenance, like fixing the wrong infrastructure or replacing perfectly good pipes because of age and material, though they are not faulty at all.

OSCON Data 2011, being held July 25-27 in Portland, Ore., is a gathering for developers who are hands-on, doing the systems work and evolving architectures and tools to manage data. (This event is co-located with OSCON.)

Save 20% on registration with the code OS11RAD

What types of data are available in a water network?

Guy Horowitz: TaKaDu does not add any new sensors, but uses existing sensors and data. Data from flow and pressure meters, Geographical information system (GIS) data, maintenance records, access control records, quality sensors, and many other types are available.

Water networks can be divided into smaller areas, dubbed district metered areas (DMAs). This approach can be helpful in monitoring water loss, but it is costly to implement and, until systems like TaKaDu, required heavy human interpretation of data. TaKaDu can take a DMA-ed network and automate most of the detection without adding new meters or sensors. Since we detect deviations from normal behavior, we typically ask for a year of history to account for all benign seasonal patterns and holiday exceptions.

What are the main techniques you use to analyze water data?

Guy Horowitz: The full explanation would be lengthy, but one example is cross-site correlation. Your neighborhood and some other neighborhood maintain a consumption relationship that holds true year-long. Why? They may have a similar demographic make-up or similar mix of residential and industrial customers. It doesn’t really matter as long as they demonstrate a similar behavior across time. Now assume your neighborhood’s consumption goes up by 10% while the “similar” neighborhood does not. That may indicate a problem, while if both increase it may indicate a warm day, or some other benign explanation. Other algorithms kick in to check all possible explanations, and if nothing explains the change, it is declared to be a problem and is classified according to its type.

Do you detect other kinds of problems apart from leaks?

Guy Horowitz: Leaks and bursts account for only about a third of the types of problems detected by TaKaDu. A majority of alerts are on inefficiencies and faults in the network setup, operation or transmission, including faulty equipment and incorrect metering. The system also gives alerts on water quality issues, energy inefficiencies and other types of faults.

What’s next for data analytics in water infrastructure management?

Guy Horowitz: We see very high interest from across the industry in smarter data-driven solutions for better management of water networks. I expect larger players to make significant moves in product development and collaboration with innovative smart-water players. The challenges posed by aging infrastructure and the growing investment gap in the water sector means that we need the best brains to start thinking about water problems rather than developing new mobile apps. The coming decade will be the decade of smarter water networks.

This interview was edited and condensed.

Related:

• Building data startups
• Interest in renewable energy could benefit data services
• Industrial ecology and big data