On March 11 Boston will join several other cities who have host conferences on the movement broadly known as NoSQL. Cassandra, CouchDB, HBase, HypergraphDB, Hypertable, Memcached, MongoDB, Neo4j, Riak, SimpleDB, Voldemort, and probably other projects as well will be represented at the one-day affair.

It's generally understood that characterizing a movement by what it's not is awkward, and it's hard to find an elevator speech to encompass all the topics of NoSQL Boston. Are these tools for "big data" problems? Usually, but sometimes even small web sites can find them useful. Are the tools meant for processing streams such as log files? Sometimes, but they can be useful for other text and data processing as well. And do they reject relational principles? Well, so you'd think--but different ones reject different principles, so even there it's hard to find commonality. (I compared them to relational databases in a blog last year.

The interviews I had with various projects leaders for this article turned up a recurring usage pattern for NoSQL. I was seeking particular domains or types of data where the tools would be useful, but couldn't see much commonality. What connects the users is that they carry out web-related data crunching, searching, and other Web 2.0 related work. I think these companies use NoSQL tools because they're the companies who understand leading-edge technologies and are willing to take risks in those areas. As the field gets better known, usage will spread.

I had a talk last week with conference organizer Eliot Horowitz, who is the founder and CTO of 10gen, the company that makes MongoDB. He let me know that the conference plans to bypass the head-scratching and launch into practical applications. The day will contain a coding session and a schema design session along with keynotes.

The resilience of open source

One question that intrigues me is why all the offerings in the NoSQL area are open source. Some have commercial add-ons, but the core technology is provided as free software. The few proprietary products and services in the market (such as Citrusleaf) get far less attention. Reasons seem to include:

• The market is currently too small. Just as most computing innovations start off in research settings, this one is being explored by people looking for solutions to their own problems, more than ways to extract a profit. Numerous in-house projects exist in this space that are not free software (Google's Map/Reduce and BigTable, for instance, and Amazon's SimpleDB and Dynamo) but they aren't commercialized either.
• Experimentation is moving too fast. Most of the projects are just a couple years old, and are rapidly adding features.
• The ROI is hard to calculate. Horowitz says, "People won't pay for anything they don't really understand yet." (Nevertheless, 10gen and other companies are commercializing the open source offerings.)
• Whatever problem an organization is trying to solve, each NoSQL offering tends to be a piece of the solution. It has to be tuned for and integrated into the organization's architecture, and combined with pieces from other places.

The projects in this conference therefore demonstrate the innovative power of free software. CouchDB and Cassandra are particularly interesting in this regard because they are community efforts more than corporate efforts. Both are Apache top-level projects. (Cassandra was just moved from the incubator to a top-level project on February 17.) CouchDB committer J. Chris Anderson tells me that the Apache community process ensures a wide range of voices are heard, leading to (of course) occasional public wrangling but a superior outcome.

The BBC and (according to Anderson) SXSW are among the users of CouchDB, CouchDB has been integrated into Ubuntu, Mozilla Messaging is basing Raindrop (their next-generation messaging platform) on CouchDB, and even mobile handset manufacturers are looking at it. (O'Reilly Media also uses CouchDB.)

I also talked to Alan Hoffman of Cloudant, which offers a CouchDB cloud service that fills in some of the gaps left by bare CouchDB (consistent hashing, partitioning, quorum, etc.). Although a couple companies offer commercial support, no single company takes responsibility for CouchDB. Its community is highly distributed. Anderson listed 10 Apache committers working for 8 different companies, and nearly 40 other people who contribute patches. Support takes place on mailing lists (roughly one thousand messages a month) and IRC channels.

Jonathan Ellis, project chair of Cassandra, calls it an "open source success story" because it went from a state of near petrification to vibrant regrowth through open sourcing. Facebook invented it and brought it to a state where it satisfied their needs. They made it open in and moved it into the Apache Incubator in 2008 but declared that they would not be doing further development. It could easily have receded into obscurity.

Ellis says that he was hired at Rackspace and asked to find a distributed data store that was fast and scaled easily; he decided on Cassandra. Soon after he became a public and enthusiastic advocate, Digg and Twitter joined Rackspace as users and developers. Having multiple QA teams test each release--particularly in very different environments--helps quality immensely. Ellis find that Eric Raymond's "many eyes" characterization of open source bug fixing applies.

Although Cassandra is found mostly as a backing store for web sites with a lot of users, Ellis thinks it would meet the needs of many academic and commercial sites, and looks forward to someone offering a cloud service based on it.

Justin Sheehy, CTO of Basho, maker of the Riak data store, told me they can confirm the typical advantages cited for open source. Developers at potential customer sites can try out the software without going through a bureaucratic procurement process, and then become internal advocates who function much more effectively than outside salespeople.

He also says that companies such as Basho offer the best of both worlds to tentative customers. The backing of a corporation means that professional services and added tools are available to go along with the product those customers buy. But because the source is open and has a community around it, those customers can feel secure that development and support will continue regardless of the fate of the originating company. 10gen, of course, plays a similar role for MongoDB and Anderson's company Couchio offers support for CouchDB. For projects that are not closely associated with the backing of one company, the Apache Foundation's sponsorship helps to ensure continuity.

What are the fault lines in the NoSQL landscape?

Naturally, the projects I've mentioned in this blog borrow ideas from each other and show tiny variations on common solutions regarding such things as B-tree storage, replication, solutions to locality of reference, etc. Experience will eventually lead to a shake-out and a convergence among surviving projects. In the meanwhile, how can you get your head around them?

We'll pause here for a word from our sponsors, letting you know that O'Reilly has published books on CouchDB and Hadoop and is developing one about MongoDB.

Horowitz offers an initial subdivision of projects based on data model (document, key-value, or tabular), a theme he explored in another interview.

Roger Magoulas, a research director with O'Reilly, further subdivides projects into those that crunch large data sets in a batch manner--such as Hadoop--and those that retrieve views of data to fulfill visitor search requests on web pages or similar tasks. He goes on to say that you can compare them on the basis of particular features, such as automatic replication, auto-sharding or partitioning, and in-memory caches.

The most comprehensive attempts I've seen to make sense of this gangly crew of projects from a feature standpoint come in a blog by Ellis and one by blog by Vineet Gupta. (Gupta's blog is labeled "Part 1" and I'd love to see more parts.) But Sheehy says the various features of the offerings interact too strongly and have too many subtle variations to fit into an easy taxonomy. "Many people try to classify the projects, everyone does it differently, and nobody gets it quite right."

Community features

So who uses these things? To take Horowitz's MongoDB again as an example, many web sites gravitate toward it because the document structure makes some things--adding fields to rows, mapping objects to fields--easier than a relational database does. A few scientific sites also use MongoDB.

Riak also has a large following among web sites and startups, but their customers also include media companies, ad networks, SMS gateways, analytics firms, and many other types of organizations.

Magoulas finds that an organization's bent is determined by the background and expertise of its developers. Programmers with lots of traditional relational database experience tend to be wary of the recent upstarts, a position reinforced by legacy investments in tools that depend on their relational database and are sometimes very expensive.

On the other hand, web programmers look for tools that conform more closely to the data structures and programming techniques they're used to, and can actually be "flummoxed" by relational database logic or abstraction layers on top of the databases. These programmers may think it intuitive to do the kinds of filtering and sorting that seem like reinventing the wheel to a traditional RDMBS programmer. Anderson likes to quote Jacob Kaplan-Moss, the creator of Django, as saying, "Django may be built for the Web, but CouchDB is built of the Web. I've never seen software that so completely embraces the philosophies behind HTTP."

10gen's consultation with MongoDB users includes asking for votes on new features. They also see a great deal of code contributions in the driver layer and adapters (sessions, logging, etc.) but not much in the core. Sheehy said the same is true of Riak: although contributions to the core are rare, half the client libraries are developed by outsiders, and many of the tools.

Rapid change is part of life for NoSQL developers. Anderson says of CouchDB, "The ancillary APIs have been evolving rapidly in preparation for our 1.0 release, which should come out in the next few months and won't differ much from today's trunk. The new APIs include authentication, authorization, details of Map/Reduce, and functions for transforming and serving JSON documents as other datatypes such as HTML or CSV." Horowitz stressed that MongoDB will roll out a lot of new features over the upcoming year.

One hundred people have signed up for NoSQL Boston so far, and more than 150 are expected. I'll be there to take it in and try to reduce it to some high-level insights for this blog.