- Decentralised Autonomous Corporations — Charlie Stross’s near-future fiction of Accelerando comes closer to reality: Malice – revenge for waking him up – sharpens Manfred’s voice. “The president of agalmic.holdings.root.184.97.AB5 is agalmic.holdings.root.184.97.201. The secretary is agalmic.holdings.root.184.D5, and the chair is agalmic.holdings.root.184.E8.FF. All the shares are owned by those companies in equal measure, and I can tell you that their regulations are written in Python. Have a nice day, now!” He thumps the bedside phone control and sits up, yawning, then pushes the do-not-disturb button before it can interrupt again. After a moment he stands up and stretches, then heads to the bathroom to brush his teeth, comb his hair, and figure out where the lawsuit originated and how a human being managed to get far enough through his web of robot companies to bug him.
- Coding is Not the New Literacy (Chris Grainger) — We build mental models of everything – from how to tie our shoes to the way macro-economic systems work. With these, we make decisions, predictions, and understand our experiences. If we want computers to be able to compute for us, then we have to accurately extract these models from our heads and record them. Writing Python isn’t the fundamental skill we need to teach people. Modeling systems is. Amen!
- Let’s Stop Laughing at Groupon (Fortune) — it is much easier to survive a valuation decline as a public company than as a private one.
- nsq — Bitly’s open sourced realtime distributed messaging platform.
Networks graphs can be used as primary visual objects with conventional charts used to supply detailed views
With Network Science well on its way to being an established academic discipline, we’re beginning to see tools that leverage it. Applications that draw heavily from this discipline make heavy use of visual representations and come with interfaces aimed at business users. For business analysts used to consuming bar and line charts, network visualizations take some getting used. But with enough practice, and for the right set of problems, they are an effective visualization model.
In many domains, networks graphs can be the primary visual objects with conventional charts used to supply detailed views. I recently got a preview of some dashboards built using Financial Network Analytics (FNA). Read more…
Analytic services are tailoring their solutions for specific problems and domains
In relatively short order Amazon’s internal computing services has become the world’s most successful cloud computing platform. Conceived in 2003 and launched in 2006, AWS grew quickly and is now the largest web hosting company in the world. With the recent addition of Kinesis (for stream processing), AWS continues to add services and features that make it an attractive platform for many enterprises.
A few other companies have followed a similar playbook: technology investments that benefit a firm’s core business, is leased out to other companies, some of whom may operate in the same industry. An important (but not well-known) example comes from finance. A widely used service provides users with clean, curated data sets and sophisticated algorithms with which to analyze them. It turns out that the world’s largest asset manager makes its investment and risk management systems available to over 150 pension funds, banks, and other institutions. In addition to the $4 trillion managed by BlackRock, the company’s Aladdin Investment Management system is used to manage1 $11 trillion in additional assets from external managers.
"Modelers have a bigger responsibility now than ever before."
People come to data science in all sorts of ways. I happen to be someone who came via finance. Trained as a mathematician, I worked first at a hedge fund and then a financial risk software company, each for about two years, starting in June 2007 and ending in February 2011. If you look at those dates again, you’ll realize I had a front row seat for the financial crisis.
I worked on a few projects in algorithmic trading with Larry Summers at the hedge fund and was invited, along with the other quants at Shaw, to see him discuss the impending doom one evening with Alan Greenspan and Robert Rubin. It honestly kind of surprised and shocked me to see how little they seemed to know, or at least admitted to knowing, about the true situation in the markets. These guys were supposed to be the experts, after all.
A math band-aid will distract us from fixing the problems that so desperately need fixing.
This piece originally appeared on Mathbabe. We’re also including Jordan Ellenberg’s counter-point to Cathy’s original post as well as Cathy’s response to Jordan. All of these pieces are republished with permission.
I just finished reading Nate Silver’s newish book, The Signal and the Noise: Why so many predictions fail – but some don’t.
The good news
First off, let me say this: I’m very happy that people are reading a book on modeling in such huge numbers – it’s currently eighth on the New York Times best seller list and it’s been on the list for nine weeks. This means people are starting to really care about modeling, both how it can help us remove biases to clarify reality and how it can institutionalize those same biases and go bad.
As a modeler myself, I am extremely concerned about how models affect the public, so the book’s success is wonderful news. The first step to get people to think critically about something is to get them to think about it at all.
Moreover, the book serves as a soft introduction to some of the issues surrounding modeling. Silver has a knack for explaining things in plain English. While he only goes so far, this is reasonable considering his audience. And he doesn’t dumb the math down.
In particular, Silver does a nice job of explaining Bayes’ Theorem. (If you don’t know what Bayes’ Theorem is, just focus on how Silver uses it in his version of Bayesian modeling: namely, as a way of adjusting your estimate of the probability of an event as you collect more information. You might think infidelity is rare, for example, but after a quick poll of your friends and a quick Google search you might have collected enough information to reexamine and revise your estimates.)
The bad news
Having said all that, I have major problems with this book and what it claims to explain. In fact, I’m angry.
It would be reasonable for Silver to tell us about his baseball models, which he does. It would be reasonable for him to tell us about political polling and how he uses weights on different polls to combine them to get a better overall poll. He does this as well. He also interviews a bunch of people who model in other fields, like meteorology and earthquake prediction, which is fine, albeit superficial.
What is not reasonable, however, is for Silver to claim to understand how the financial crisis was a result of a few inaccurate models, and how medical research need only switch from being frequentist to being Bayesian to become more accurate. Read more…
Tablet table, Google contest for students, watch your incentives, research on web search abandonment.
- Latest Tablets (Luke Wroblewski) — table showing the astonishing variety of tablets released in the last two months.
- Google Code-In Contest for High Schoolers — an international contest introducing 13-17 year old pre-university students to the world of open source software development. The goal of the contest is to give students the opportunity to explore the many types of projects and tasks involved in open source software development. (via Andy Oram)
- Watch Your Incentives — NASDAQ added two new incentives programs, and robotraders responded. On November 1st, there were 369 seconds where the number of quotes in BAC exceeded 17,000; a total of 6.6 million quotes. During those seconds, only 1,879 trades executed. Between market open (9:30am) and 12:45, BAC had 7.8 million quotes and 116,000 trades. Which means 85% of all BAC quotes occurred in those 369 seconds. Which means it is likely that one algo from one firm (all of this quote spam is from Nasdaq) is responsible for 85% of all canceled orders in BAC. (via Robert O’Brien)
- Predicting Web Search Abandonment Rationales (PDF) — Microsoft Research paper on how to predict why people cease to search or to click through on the search results. I’m really impressed by how well they could distinguish “bah, the Internet doesn’t have it, I’m giving up” from “oh, the answer was in a search result snippet, my work here is done.” (via Mark Alen)
Crowdsourcing Flights, Teaching Programming, Redeploying Finance Engineers, and Recognising Cat Faces
- Flightfox — Real people compete to find you the best flights. Crowdsourcing beating algorithms …. (via NY Times)
- Code Monster (Crunchzilla) — a fun site for parents to learn to program with their kids. Loving seeing so much activity around teaching kids to program. (via Greg Linden)
- Telling People to Leave Finance (Cathy O’Neil) — There’s an army of engineers in finance that could be putting their skills to use with actual innovation rather than so-called financial innovation.
A few early and broad questions in our exploration of NYC's startup community.
Since the crisis of 2008 New York City’s massive financial sector — the city’s richest economic engine, once seen to have unlimited potential for growth — has languished. In the meantime, attention has turned to its nascent startup sector, home to Foursquare, Tumblr, 10gen, Etsy and Gilt, where VC investment has surged even as it’s been flat in other big U.S. tech centers (PDF).
I’ve started to poke around the tech community here with a view toward eventually publishing a paper on the rise of New York’s startup scene. In my initial conversations, I’ve come up with a few broad questions I’ll focus on, and I’d welcome thoughts from this blog’s legion of smart readers on any of these.
- How many people in New York’s startup community came from finance, and under what conditions did they make the move? In 2003, Google was a five-year-old, privately-held startup and Bear Stearns was an 80-year-old pillar of the financial sector. Five years later, Google was a pillar of the technical economy and among the world’s biggest companies; Bear Stearns had ceased to exist. Bright quantitatively-minded people who might have pursued finance for its stability and lucre now see that sector as unstable and not necessarily lucrative; its advantage over the technology sector in those respects has disappeared. Joining a 10-person startup is very different from taking a job at Google, but the comparative appeal of the two sectors has dramatically shifted.
- To what degree have anchor institutions played a role in the New York startup scene? The relationship between Stanford University and Silicon Valley is well-documented; I’d like to figure out who’s producing steady streams of bright technologists in New York. Google’s Chelsea office, opened in 2006, now employs close to 3,000 people, and its alumni include Dennis Crowley, founder of Foursquare. That office is now old enough that it can generate a high volume of spin-offs as Googlers look for new challenges. And Columbia and NYU (and soon a Cornell-Technion consortium) have embraced New York’s startup community.
Putting high-frequency trading into perspective.
Technology is critical to today’s financial markets. It’s also surprisingly controversial. In most industries, increasing technological involvement is progress, not a problem. And yet, people who believe that computers should drive cars suddenly become Luddites when they talk about computers in trading.
There’s widespread public sentiment that technology in finance just screws the “little guy.” Some of that sentiment is due to concern about a few extremely high-profile errors. A lot of it is rooted in generalized mistrust of the entire financial industry. Part of the problem is that media coverage on the issue is depressingly simplistic. Hyperbolic articles about the “rogue robots of Wall Street” insinuate that high-frequency trading (HFT) is evil without saying much else. Very few of those articles explain that HFT is a catchall term that describes a host of different strategies, some of which are extremely beneficial to the public market.
I spent about six years as a trader, using automated systems to make markets and execute arbitrage strategies. From 2004-2011, as our algorithms and technology became more sophisticated, it was increasingly rare for a trader to have to enter a manual order. Even in 2004, “manual” meant instructing an assistant to type the order into a terminal; it was still routed to the exchange by a computer. Automating orders reduced the frequency of human “fat finger” errors. It meant that we could adjust our bids and offers in a stock immediately if the broader market moved, which enabled us to post tighter markets. It allowed us to manage risk more efficiently. More subtly, algorithms also reduced the impact of human biases — especially useful when liquidating a position that had turned out badly. Technology made trading firms like us more profitable, but it also benefited the people on the other sides of those trades. They got tighter spreads and deeper liquidity.