- Update on indie.vc — We’ve worked with the team at Cooley to create an investment instrument that has elements of both debt and equity. Debt in that we will not be purchasing equity initially, but, unlike debt, there is no maturity date, no collateralization of assets and no recourse if it’s never paid back. The equity element will only become a factor if the participating company chooses to raise a round of financing or sell out to an acquiring company. We don’t have a clever acronym or name for this instrument yet, but I’m sure we’ll come up with something great.
- How Nathan Barley Came True (Guardian) — if you haven’t already seen Nathan Barley, you should. It’s by the guy who did Black Mirror, and it’s both awful and authentic and predictive and retro and … painfully accurate about the horrors of our Internet/New Media industry. (via BoingBoing)
- Trust Engineers (Radio Lab) — Facebook has a created a laboratory of human behavior the likes of which we’ve never seen. We peek into the work of Arturo Bejar and a team of researchers who are tweaking our online experience, bit by bit, to try to make the world a better place. Radio show of goodness. (via Flowing Data)
- DARPA’S Haptix Project — The goal of the HAPTIX program is to provide amputees with prosthetic limb systems that feel and function like natural limbs, and to develop next-generation sensorimotor interfaces to drive and receive rich sensory content from these limbs. Today it’s prosthetic limbs for amputees, but within five years it’ll be augmented ad-driven realities for virtual currency ambient social recommendations.
The O'Reilly Data Show Podcast: Gary Kazantsev on how big data and data science are making a difference in finance.
Having started my career in industry, working on problems in finance, I’ve always appreciated how challenging it is to build consistently profitable systems in this extremely competitive domain. When I served as quant at a hedge fund in the late 1990s and early 2000s, I worked primarily with price data (time-series). I quickly found that it was difficult to find and sustain profitable trading strategies that leveraged data sources that everyone else in the industry examined exhaustively. In the early-to-mid 2000s the hedge fund industry began incorporating many more data sources, and today you’re likely to find many finance industry professionals at big data and data science events like Strata + Hadoop World.
During the latest episode of the O’Reilly Data Show Podcast, I had a great conversation with one of the leading data scientists in finance: Gary Kazantsev runs the R&D Machine Learning group at Bloomberg LP. As a former quant, I wanted to know the types of problems Kazantsev and his group work on, and the tools and techniques they’ve found useful. We also talked about data science, data engineering, and recruiting data professionals for Wall Street. Read more…
Networks graphs can be used as primary visual objects with conventional charts used to supply detailed views
With Network Science well on its way to being an established academic discipline, we’re beginning to see tools that leverage it. Applications that draw heavily from this discipline make heavy use of visual representations and come with interfaces aimed at business users. For business analysts used to consuming bar and line charts, network visualizations take some getting used. But with enough practice, and for the right set of problems, they are an effective visualization model.
In many domains, networks graphs can be the primary visual objects with conventional charts used to supply detailed views. I recently got a preview of some dashboards built using Financial Network Analytics (FNA). Read more…
Analytic services are tailoring their solutions for specific problems and domains
In relatively short order Amazon’s internal computing services has become the world’s most successful cloud computing platform. Conceived in 2003 and launched in 2006, AWS grew quickly and is now the largest web hosting company in the world. With the recent addition of Kinesis (for stream processing), AWS continues to add services and features that make it an attractive platform for many enterprises.
A few other companies have followed a similar playbook: technology investments that benefit a firm’s core business, is leased out to other companies, some of whom may operate in the same industry. An important (but not well-known) example comes from finance. A widely used service provides users with clean, curated data sets and sophisticated algorithms with which to analyze them. It turns out that the world’s largest asset manager makes its investment and risk management systems available to over 150 pension funds, banks, and other institutions. In addition to the $4 trillion managed by BlackRock, the company’s Aladdin Investment Management system is used to manage1 $11 trillion in additional assets from external managers.
"Modelers have a bigger responsibility now than ever before."
People come to data science in all sorts of ways. I happen to be someone who came via finance. Trained as a mathematician, I worked first at a hedge fund and then a financial risk software company, each for about two years, starting in June 2007 and ending in February 2011. If you look at those dates again, you’ll realize I had a front row seat for the financial crisis.
I worked on a few projects in algorithmic trading with Larry Summers at the hedge fund and was invited, along with the other quants at Shaw, to see him discuss the impending doom one evening with Alan Greenspan and Robert Rubin. It honestly kind of surprised and shocked me to see how little they seemed to know, or at least admitted to knowing, about the true situation in the markets. These guys were supposed to be the experts, after all.
A math band-aid will distract us from fixing the problems that so desperately need fixing.
This piece originally appeared on Mathbabe. We’re also including Jordan Ellenberg’s counter-point to Cathy’s original post as well as Cathy’s response to Jordan. All of these pieces are republished with permission.
I just finished reading Nate Silver’s newish book, The Signal and the Noise: Why so many predictions fail – but some don’t.
The good news
First off, let me say this: I’m very happy that people are reading a book on modeling in such huge numbers – it’s currently eighth on the New York Times best seller list and it’s been on the list for nine weeks. This means people are starting to really care about modeling, both how it can help us remove biases to clarify reality and how it can institutionalize those same biases and go bad.
As a modeler myself, I am extremely concerned about how models affect the public, so the book’s success is wonderful news. The first step to get people to think critically about something is to get them to think about it at all.
Moreover, the book serves as a soft introduction to some of the issues surrounding modeling. Silver has a knack for explaining things in plain English. While he only goes so far, this is reasonable considering his audience. And he doesn’t dumb the math down.
In particular, Silver does a nice job of explaining Bayes’ Theorem. (If you don’t know what Bayes’ Theorem is, just focus on how Silver uses it in his version of Bayesian modeling: namely, as a way of adjusting your estimate of the probability of an event as you collect more information. You might think infidelity is rare, for example, but after a quick poll of your friends and a quick Google search you might have collected enough information to reexamine and revise your estimates.)
The bad news
Having said all that, I have major problems with this book and what it claims to explain. In fact, I’m angry.
It would be reasonable for Silver to tell us about his baseball models, which he does. It would be reasonable for him to tell us about political polling and how he uses weights on different polls to combine them to get a better overall poll. He does this as well. He also interviews a bunch of people who model in other fields, like meteorology and earthquake prediction, which is fine, albeit superficial.
What is not reasonable, however, is for Silver to claim to understand how the financial crisis was a result of a few inaccurate models, and how medical research need only switch from being frequentist to being Bayesian to become more accurate. Read more…
Tablet table, Google contest for students, watch your incentives, research on web search abandonment.
- Latest Tablets (Luke Wroblewski) — table showing the astonishing variety of tablets released in the last two months.
- Google Code-In Contest for High Schoolers — an international contest introducing 13-17 year old pre-university students to the world of open source software development. The goal of the contest is to give students the opportunity to explore the many types of projects and tasks involved in open source software development. (via Andy Oram)
- Watch Your Incentives — NASDAQ added two new incentives programs, and robotraders responded. On November 1st, there were 369 seconds where the number of quotes in BAC exceeded 17,000; a total of 6.6 million quotes. During those seconds, only 1,879 trades executed. Between market open (9:30am) and 12:45, BAC had 7.8 million quotes and 116,000 trades. Which means 85% of all BAC quotes occurred in those 369 seconds. Which means it is likely that one algo from one firm (all of this quote spam is from Nasdaq) is responsible for 85% of all canceled orders in BAC. (via Robert O’Brien)
- Predicting Web Search Abandonment Rationales (PDF) — Microsoft Research paper on how to predict why people cease to search or to click through on the search results. I’m really impressed by how well they could distinguish “bah, the Internet doesn’t have it, I’m giving up” from “oh, the answer was in a search result snippet, my work here is done.” (via Mark Alen)
Crowdsourcing Flights, Teaching Programming, Redeploying Finance Engineers, and Recognising Cat Faces
- Flightfox — Real people compete to find you the best flights. Crowdsourcing beating algorithms …. (via NY Times)
- Code Monster (Crunchzilla) — a fun site for parents to learn to program with their kids. Loving seeing so much activity around teaching kids to program. (via Greg Linden)
- Telling People to Leave Finance (Cathy O’Neil) — There’s an army of engineers in finance that could be putting their skills to use with actual innovation rather than so-called financial innovation.