Find emergent properties and solutions to new computing problems with graphs
Graph databases haven’t made the news much because, I think, they don’t fit in convenient categories. They certainly aren’t the relational databases we’re all familiar with, nor are they the arbitrary keys and values provided by many NoSQL stores. But in a highly connected world–where it’s not what you know but whom you know–it makes intuitive sense to arrange our knowledge as nodes and edges.
Ted Nelson, inventor of the hyperlink, recognized the power of viewing life in graphs. After the implosion of his historic Xanadu project, he embarked on a graph database tool called ZigZag. The most modern instantiations of graphs–the Neo4j store and the Alchemy.js tool for interactively visualizing graphs–were well represented this year at O’Reilly’s Open Source convention.
From tiny satellites to young programmers to reasoned paranoia, here are key talks from OSCON 2014.
Experts and advocates from across the open source world assembled in Portland, Ore. this week for OSCON 2014. Below you’ll find a handful of keynotes and interviews from the event that we found particularly notable.
How tiny satellites and fresh imagery can help humanity
Will Marshall of Planet Labs outlines a vision for using small satellites to provide daily images of the Earth.
Can education and peer review keep a huge open source project on track?
When does a software project grow to the point where one must explicitly think about governance? The term “governance” is stiff and gawky, but doing it well can carry a project through many a storm. Over the past couple years, the crucial OpenStack project has struggled with governance at least as much as with the technical and organizational issues of coordinating inputs from thousands of individuals and many companies.
A major milestone was the creation of the OpenStack Foundation, which I reported on in 2011. This event successfully started the participants’ engagement with the governance question, but it by no means resolved it. This past Monday, I attended some of the Open Cloud Day at O’Reilly’s Open Source convention, and talked to a lot of people working for or alongside the OpenStack Foundation about getting contributors to work together successfully in an open community. Read more…
Variations in Test-Driven Development
“Red-Green-Refactor” is a familiar slogan from test-driven development (TDD), describing a popular approach to writing software. It’s been both popular and controversial since the 2000’s (see the recent heated discussions between David Hansson, Bob Martin, and others). I find that it’s useful but limiting. Here I’ll describe some interesting exceptions to the rule, which have expanded the way I think about tests.
The standard three-step cycle goes like this. After choosing a small improvement, which can be either a feature or a bug fix, you add a failing test which shows that the improvement is missing (“Red”); add production code to make the test pass (“Green”); and clean up the production code while making sure the tests still pass (“Refactor”). It’s a tight loop with minimal changes at each step, so you’re never far from code that runs and has good test coverage.
By the way, to simplify things, I’ll just say “tests” and be vague about whether they’re technically “unit tests”, “specs,” “integration tests,” or “functional tests”; the main thing is that they’re written in code and they run automatically.
Red-Green-Refactor is a very satisfying rhythm when it works. Starting from the test keeps the focus on adding value, and writing a test forces you to clarify where you want to go. Many people say it promotes clean design: it’s just easier to write tests when you have well-separated modules with reasonable interfaces between them. My personal favorite part, though, is not the Red but the Refactor: the support from tests allows you to clean things up with confidence, and worry less about regressions.
Now for the exceptions. Read more…
PayPal has gone through a cultural transformation with radical transparency as a cornerstone of the plan.
Three years ago, PayPal was growing exponentially, staying profitable and was considered the most successful online payments company in the world. This should have been the recipe of a company that was attracting top talent across the globe, and keeping their core engineers happy, thriving, and innovative. But, at the time, the PayPal engineering team wasn’t where they needed to be to stay ahead of the curve — they didn’t have the process, the tools, or the resources to extend their talent and stay engaged in creating amazing products and services.
Leadership had encouraged the formation of engineering silos to “concentrate expertise,” but this made it incredibly challenging to get things done. At the same time, popular services such as Google and Amazon were raising the bar for everybody. All businesses — not just software-focused businesses — needed to have websites (and mobile apps) that were snazzy and responsive in addition to being reliable. PayPal engineering needed to push the proverbial envelope to stay competitive in a fierce and unrelenting industry landscape.
For PayPal, the transformation started at the edge of the stack. The Kraken project, which was started by an internal team to support a new checkout system, proved that an open source platform could reduce time to market and still perform at scale. This was achieved largely in spite of the silo culture that ran rampant and tended to restrict innovation and creativity. Support from senior management and perception of less risk at the edge of the stack helped the project and ultimately unleashed a gold rush of interest in repeating the win with releases of internally developed improvements to other open source projects. When I came into PayPal, I received an avalanche of mail from teams who wanted to “open source something.”