In a recent interview, Joe Hellerstein, a professor in the UC Berkeley computer science department, talked about the disconnect between open source innovation and development. The problem, he said, doesn’t lie with funding, but with engineering and professional development:
As I was coming up as a student, really interesting open source was coming out of universities. I’m thinking of things like the Ingres and Postgres database projects at Berkeley and the Mach operating system at Carnegie Mellon. These are things that today are parts of commercial products, but they began as blue-sky research. What has changed now is there’s more professionally done open source. It’s professional, but it’s further disconnected from research.
A lot of the open source that’s very important is really “me-too” software — so Linux was a clone of Unix, and Hadoop is a clone of Google’s MapReduce. There’s a bit of a disconnect between the innovation side, which the universities are good at, and the professionalism of open source that we expect today, which the companies are good at. The question is, can we put those back together through some sort of industrial-academic partnership? I’m hopeful that can be done, but we need to change our way of business.
Hellerstein pointed to the MADlib project being conducted between his group at Berkeley and the project sponsor EMC Greenplum as an example of a new partnership model that could close the gap between innovation and development.
Our sponsor would have been happy to donate money to my research funds, but I said, “You know, what I really need is engineering time.”
The thing I cannot do on campus is run a professional engineering shop. There are no career incentives for people to be programmers at the university. But a company has processes and expertise, and they can hire really good people who have a career path in the company. Can we find an arrangement where those people are working on open source code in collaboration with the people at the university?
It’s a different way of doing research funding. The company’s contributions are not financial. The contributions are in engineering sweat. It’s an interesting experiment, and it’s going well so far.
In the interview Hellerstein also discusses MAD data analysis and where we are in the industrial revolution of data. The full interview is available in the following video: