Subscribe to the O’Reilly Data Show Podcast to explore the opportunities and techniques driving big data and data science.
In this episode of the O’Reilly Data Show, I sat down with Vasant Dhar, a professor at the Stern School of Business and Center for Data Science at NYU, founder of SCT Capital Management, and editor-in-chief of the Big Data Journal (full disclosure: I’m a member of the editorial board). We talked about the early days of AI and data mining, and recent applications of data science to financial investing and other domains.
Dhar’s first steps in applying machine learning to finance
I joke with people, I say, ‘When I first started looking at finance, the only thing I knew was that prices go up and down.’ It was only when I actually went to Morgan Stanley and took time off from academia that I learned about finance and financial markets. … What I really did in that initial experiment is I took all the trades, I appended them with information about the state of the market at the time, and then I cranked it through a genetic algorithm and a tree induction algorithm. … When I took it to the meeting, it generated a lot of really interesting discussion. … Of course, it took several months before we actually finally found the reasons for why I was observing what I was observing.
Robots as decision makers
The general research question I really ask is when do computers make better decisions than humans? That’s really sort of the core question. … I’ve applied it to finance, but there are other areas. I’m involved in a project on education, and one might ask the same thing. When do computers make better teachers than humans? It’s an equally interesting question. … Should you trust your money to a robot? The flip side of that question is when do computers make better decisions than humans?
…
One of the things I did was to break up the investment landscape into three different types of holding periods. On the one hand, you have high-frequency trading, and on the other extreme, you have very long-term investing. In high-frequency trading, your holding periods are sort of minutes to a day. In very long-term investing, your holding periods are months to years, that Warren Buffett style of investing. Then there’s sort of a space in the middle, which is the part I find most interesting, where there’s a lot of action, which is sort of days to weeks holding period. … The strategy one uses for these different horizons tends to be very different. In the high-frequency trading space, for example, humans don’t really stand a chance against computers, there’s just so much information.
Related resources:
- The race to build ‘‘big data machines’’ in financial investing
- Minds and machines – Humans where they’re best, robots for the rest: Adam Marcus’ presentation at Hardcore Data Science (Strata + Hadoop World NYC 2015)
- Fast data is the key to next-gen capital and risk management
Image via the Internet Archive on Wikimedia Commons.