How Big Data Impacts Analytics

Egon Willighagen [2009-04-28 04:49 AM]

ORF++

Falafulu Fisi [2009-04-28 09:24 AM]

LinkedIn's data is pretty much static and there isn't much pre-processing involved unless I am wrong here. It is a huge dataset, no doubt, but it is not a fast evolving dataset that complicates analysis such as data from scientific experimentation (particle physics, engineering electro-magnetic, fluid dynamic, etc...) or from financial markets, where trading is being registered every few milliseconds or so (worldwide) where the analytics is very complex including its pre-processing steps and it needs to be dynamic & realtime. This means that financial models built using historical data upto the last 10 minutes ago (ie, T = t-10) , would be different from models built using historical data upto this very moment, which is now (ie, T = t). In comparison to LinkedIn, the analytic model built yesterday (or even last week) is pretty much the same as if the model is build today. This is due to the almost static nature of LinkedIn new subscribers. Sure, there are new people joining everyday, but it is unnoticeable. Google data is also an evolving dataset, since there is always a new document that pops up all the time somewhere in the world. They (Google) needs to update their dataset (every hour or so, perhaps even longer) for PageRank to re-crunch so that it is recent.

Finally, a question to DJ Patil. Is it possible that you can give us readers here about the sort of analytics that LinkedIn is doing? What sorts of metrics are you looking at or analyze? Just curious, that's all.

A. Aramburu [2009-04-28 01:56 PM]

Could you recommend any books on Analytics for business?

Jay Emerson [2009-04-29 10:14 AM]

Not only are big-data analytic infrastructures like Hadoop now available in the open-source world, but the analytic methods themselves are available too.

R (a statistical programming environment, http://www.r-project.org) is the pre-eminent open source project for statistical analysis. There are more than a few of us in the R community working on Big Data projects. My graduate student and I offer a library of functions in R to help scale those analytics to very large data sets (http://cran.r-project.org/web/packages/bigmemory/) using shared
memory and (optionally) file-backed matrices for larger-than-RAM data sets.

In addition, R has just been integrated with Hadoop by Saptarshi Guha (http://ml.stat.purdue.edu/rhipe), and the commercial open-source company REvolution Computing (whom I consult for) offers large-scale
distributed computing tools for R
(http://www.revolution-computing.com/products/parallel-r.php).

I completely agree with DJ Patil that with these open-source tools available (and increasingly easy to use), we're set to see an explosion in their use for "analytics." LinkedIn, FaceBook and
Google are already using open-source tools to analyze their data -- see http://dataspora.com/blog/predictive-analytics-using-r.

Falafulu Fisi [2009-04-29 01:16 PM]

Jay Emmerson, just a passing comment. R started here in New Zealand (University of Auckland) by Prof. Ross Ihaka at the Statistics Department. I know Prof. Ihaka (a former lecturer of mine).

Falafulu Fisi [2009-04-29 01:26 PM]

Jay Emmerson said...
...but the analytic methods themselves are available too.

Also another popular Java open source data-mining project named WEKA, that has been made available for over a decade here in New Zealand at University of Waikato. It is still growing, because there are lots of contributors to it.

Andrei Lopatenko [2009-04-29 02:44 PM]

Do not forget about the huge number of open source implementation of inference ("analytics") tools like libsvm (perfect integration with Python) and many other svm implementation (in C, Java, R packages, Matlab packages), BUGS (gibbs sampling based Bayesian inference), Apache's Mahout (they are trying to build scalable stat inference algorithm)
I believe almost any standard method of statistical inference is implemented as open source nowdays
Huge amounts of data became available to the public (recent Amazon initiative, Zillow data API, Yelp data API etc)