An ESB for the Web?

I spend a great deal of my time encouraging “enterprise people” to think more like “web people.” Focus on adoption, use platforms to enable emergent capability, build the “generative enterprise,” and that sort of thing.

So, imagine my surprise when I saw the web acting a bit like the enterprise with the launch of Gnip.

As the web moves toward a network of widespread transactional API’s, each with it’s own vocabulary, it is starting to look a lot like a legacy enterprise writ large or maybe like an industry eco-system. So we shouldn’t be surprised to see web developers turning to solutions that their enterprise colleagues would find familiar.

Anyone who has spent more than five minutes in the enterprise world talking about SOA in the last five years (or spent time building “trading platforms” for industry consortiums prior to that) has probably drawn a picture on a whiteboard that looks something like this (see, almost identical):

Whether you have integrated line of business applications inside the enterprise or connected trading partners within an industry, that N squared connection problem will resonate with your experience. Webs of poorly documented point-to-point integrations are expensive to build, expensive to maintain, and impossibly brittle when the business changes.

And now the N squared problem seems like it might be beginning to resonate with web developers too now that they have to integrate to an ever growing population of API’s. Plus, on the web, the additional limitations of a port 80 based infrastructure add to the nightmare by throwing the expense of constant API polling into the mix.

So, what to do?

In the enterprise space you would probably define an enterprise vocabulary, build a bunch of services that conform to it (or buy your applications from vendors that provide them), and then hook them all together through your Enterprise Service Bus (ESB) (feel free to argue that last point if you are in a religious frame of mind, some people love them, some people hate them).

ESB’s by definition support web services interfaces, provide translation services, and process orchestration on top of a message routing backbone. They usually come from vendors that probably used to sell Message Oriented Middleware (MOM) (of both the store and forward and pub/sub variety), Application Servers, Enterprise Application Integration (EAI), and even Export Transform and Load (ETL) software. There are also a growing stable of open source versions built on standards like Java Business Integration (JBI) – which is a poorly chosen name if there ever was one. If you don’t like buying or managing middleware, there are even ESB’s offered as a service.

If you are trying to deal with the same problem across your firewalls (i.e. with your trading partners) you may sign up with domain or industry specific trading platforms such as Elemica or Exostar that are basically big managed ESB’s that support an industry specific set of documents and translations. They may also deal with the problem of different networks and bindings (e.g. I use EDI but my trading partners use XML over port 80).

Which brings me back to Gnip as the ESB for the web.

When I first saw this drawing on their web site I immediately thought “Cool, I bet they are using a JBI backbone with a service engine for translation and a bunch of binding components to deal with XMPP, HTTP, SIP, RSS, and etc.” Because I come from the enterprise space this seems like a natural use case for an ESB, and for JBI in particular. Barring “web scale” issues it seemed like a no brainer.

I was curious to find out if that’s what they actually did and Jud Valeski, Gnip’s CTO, was kind enough to spend some time on the phone with me. As it turns out that isn’t how they are building it for reasons that are partly path dependent and partly related to scaling, but I think Jud would agree that conceptually what they are doing is very much like an ESB for the web (conveniently, I’m posting this as Jud goes off grid for three days. I’m hoping he’ll comment when he comes back). They are essentially creating a messaging bus that can provide schema translations with the added benefit of push / pull impedance matching. If REST is a web service (or close enough), then Gnip’s a managed ESB.

It will be interesting to see if developers view Gnip as a must have for dealing with their N squared pain or whether it takes more than a few bouts with legacy end-point migration to put the hurt on. I’m also interested to see if Gnip doesn’t catalyze more interest in community-wide vocabulary development. It looks a lot like an industry trading platform and by force of habit and gravitational pull those tend to facilitate shared vocabularies.

Also, the web being the web (and not an enterprise), It seems that there is a growing trend towards the use of XMPP as a generic XML routing bus, a role that makes it look suspiciously like an ESB. It strikes me as an example of “use what you have / use what you know” and I wonder if the developers that are using it that way would use projects like ServiceMix or OpenESB instead if they were coming from a different background.

One last note. You may be thinking “If message oriented middleware is the backbone of many ESB’s, why isn’t Gnip using Amazon’s SQS as the foundation for the Web’s ESB?” After all, SQS is essentially a simple web-friendly message bus. The simple answer is latency. SQS has performance characteristics more like store-and-forward-style MOM than like pub/sub MOM. Because of that it is more suitable for use cases that need guaranteed delivery but that can support average latencies on the order of one second (and may be high as ten seconds). Follow the comments in this post for more detail.

tags: , , , ,