An ESB for the Web?

I spend a great deal of my time encouraging “enterprise people” to think more like “web people.” Focus on adoption, use platforms to enable emergent capability, build the “generative enterprise,” and that sort of thing.

So, imagine my surprise when I saw the web acting a bit like the enterprise with the launch of Gnip.

As the web moves toward a network of widespread transactional API’s, each with it’s own vocabulary, it is starting to look a lot like a legacy enterprise writ large or maybe like an industry eco-system. So we shouldn’t be surprised to see web developers turning to solutions that their enterprise colleagues would find familiar.

Anyone who has spent more than five minutes in the enterprise world talking about SOA in the last five years (or spent time building “trading platforms” for industry consortiums prior to that) has probably drawn a picture on a whiteboard that looks something like this (see, almost identical):

Whether you have integrated line of business applications inside the enterprise or connected trading partners within an industry, that N squared connection problem will resonate with your experience. Webs of poorly documented point-to-point integrations are expensive to build, expensive to maintain, and impossibly brittle when the business changes.

And now the N squared problem seems like it might be beginning to resonate with web developers too now that they have to integrate to an ever growing population of API’s. Plus, on the web, the additional limitations of a port 80 based infrastructure add to the nightmare by throwing the expense of constant API polling into the mix.

So, what to do?

In the enterprise space you would probably define an enterprise vocabulary, build a bunch of services that conform to it (or buy your applications from vendors that provide them), and then hook them all together through your Enterprise Service Bus (ESB) (feel free to argue that last point if you are in a religious frame of mind, some people love them, some people hate them).

ESB’s by definition support web services interfaces, provide translation services, and process orchestration on top of a message routing backbone. They usually come from vendors that probably used to sell Message Oriented Middleware (MOM) (of both the store and forward and pub/sub variety), Application Servers, Enterprise Application Integration (EAI), and even Export Transform and Load (ETL) software. There are also a growing stable of open source versions built on standards like Java Business Integration (JBI) – which is a poorly chosen name if there ever was one. If you don’t like buying or managing middleware, there are even ESB’s offered as a service.

If you are trying to deal with the same problem across your firewalls (i.e. with your trading partners) you may sign up with domain or industry specific trading platforms such as Elemica or Exostar that are basically big managed ESB’s that support an industry specific set of documents and translations. They may also deal with the problem of different networks and bindings (e.g. I use EDI but my trading partners use XML over port 80).

Which brings me back to Gnip as the ESB for the web.

When I first saw this drawing on their web site I immediately thought “Cool, I bet they are using a JBI backbone with a service engine for translation and a bunch of binding components to deal with XMPP, HTTP, SIP, RSS, and etc.” Because I come from the enterprise space this seems like a natural use case for an ESB, and for JBI in particular. Barring “web scale” issues it seemed like a no brainer.

I was curious to find out if that’s what they actually did and Jud Valeski, Gnip’s CTO, was kind enough to spend some time on the phone with me. As it turns out that isn’t how they are building it for reasons that are partly path dependent and partly related to scaling, but I think Jud would agree that conceptually what they are doing is very much like an ESB for the web (conveniently, I’m posting this as Jud goes off grid for three days. I’m hoping he’ll comment when he comes back). They are essentially creating a messaging bus that can provide schema translations with the added benefit of push / pull impedance matching. If REST is a web service (or close enough), then Gnip’s a managed ESB.

It will be interesting to see if developers view Gnip as a must have for dealing with their N squared pain or whether it takes more than a few bouts with legacy end-point migration to put the hurt on. I’m also interested to see if Gnip doesn’t catalyze more interest in community-wide vocabulary development. It looks a lot like an industry trading platform and by force of habit and gravitational pull those tend to facilitate shared vocabularies.

Also, the web being the web (and not an enterprise), It seems that there is a growing trend towards the use of XMPP as a generic XML routing bus, a role that makes it look suspiciously like an ESB. It strikes me as an example of “use what you have / use what you know” and I wonder if the developers that are using it that way would use projects like ServiceMix or OpenESB instead if they were coming from a different background.

One last note. You may be thinking “If message oriented middleware is the backbone of many ESB’s, why isn’t Gnip using Amazon’s SQS as the foundation for the Web’s ESB?” After all, SQS is essentially a simple web-friendly message bus. The simple answer is latency. SQS has performance characteristics more like store-and-forward-style MOM than like pub/sub MOM. Because of that it is more suitable for use cases that need guaranteed delivery but that can support average latencies on the order of one second (and may be high as ten seconds). Follow the comments in this post for more detail.

tags: , , , ,
  • Jim, this seems like a stretch. This is much too domain specific to be useful outside of Data Portability.

    But more importantly, I’m not even sure that the Web even needs an ESB. Your point about web guys and enterprise guys thinking differently is the root of the matter.

  • After over a decade of enterprise work (yes, I’ve drawn that diagram, too), I wish I could remember a single example of an ESB-like thingy that lived up to any of these promises. It’s one of those good ideas on paper that doesn’t scale to a working implementation.

    First, it’s never really an n^2 problem in the enterprise, because most systems need to talk to only one or two other systems — there are usually only a few systems that everyone needs to talk to.

    Second, the small amount of data that is widely useful can generally be translated to a common format (sometimes even RSS) using adapters that live close to the systems and can be maintained by the same dev teams, avoiding a lot of communication and political problems.

  • +1 to David. I’ve done quite a lot of enterprise EAI/ESB/SOA architectures and my experience is that the two worlds (enterprise and web) are so different in regards to requirements and capabilities that using the same tools in both worlds rarely makes sense.

  • wcyee

    My experience is closer to Dave Megginson’s. ESBs always sounds great during the integration meetings and sales pitch from various vendors, but fall flat at the end of the day. The problem was never as severe as imagined (

  • wcyee

    Hmm. Not sure why it truncated my comment, but here’s one more try:

    My experience is closer to Dave Megginson’s. ESBs always sounds great during the integration meetings and sales pitch from various vendors, but fall flat at the end of the day. The problem was never as severe as imagined (<<n^2), but now everything *must* go through the ESB. Now you’ve got another problem and it’s a constant effort shoehorning these things and solving one-off problems because it’s never as easy or generic as advertised. Essentially, the only benefits realized are consulting revenues for ESB vendors and their partners.

  • I can’t help but think like the previous comments. There’s something inherently wrong with Gnip protocol bridge. It just ain’t that complex.

    And, more importantly the “web” is just an interface layer at the end of the day. Yes, I know HTTP is the transport – but, from the application perspective the web isn’t that far removed from the Enterprise. Portal anyone?

    The reality is that almost all web frameworks provide those capabilities, or have plugin architectures to support the extensibility. Let’s look at Ruby on Rails for a second. Out of the box, Rails (which implements a ReST architecture) provides RSS/Atom/HTML/JSON/etc. interface to the same data; and, that interface is instantiated by URL. Protocols such as Comet/XMPP are unique, specific purpose tools to interact with other web-ish systems (chat, etc.). It is just too easy to integrate these things into Rails to require a “custom” bus/bridge.

    As an Enterprise guy I’m not buying it. As a web guy all I see is an API aggregator. Mashup anyone? Nothing new here…just the marketing spin.

  • +1 more.

    Whatever it is that Gnip is doing, it’s either wrong or too early. The kinds of translations that diagram describes are never a clean fit, and the centralization offered just sound like a classic SPOF. The latency issue re: SQS sounds like kind of a cop out, these guys are just not providing a service that sounds like it needs blazing fast processing times.

  • Jim Stogdill

    Really interesting… I didn’t expect this post to become a referendum on the utility of ESB’s though I guess I’m really not surprised. I was really sort of trying to use Gnip as a foil for the question: “As the web exposes more API’s, and those API’s are beginning to be consumed in a configuration that looks a little bit like that classic N^2 picture, is the PATTERN that ESB’s were intended to address in the enterprise emerging on the web? And if yes, how (or should) it be managed?”

    So, are people beginning to see that pattern and the need for solutions that deal with point-to-point integration proliferation?”

    p.s. @wcyee sorry about the comment issue. I’ve pointed out the problem to the site administrator.

  • Dennis Linnell

    Looking at the Gnip architecture, I’d say we’ve seen this movie before. Starting with the Tower of Babel of biblical origins, we progressed to things like airline hubs, which were implemented entirely for the industry’s convenience, not for the customer’s. ESBs follow the same pattern, and, like that annoying airline hub, are likely to endure.

    In the 1980s and 90s we had SoftSwitch. It was a software hub that translated among myriad email formats. Back then, the email format Tower of Babel seemed a huge problem, but standards transformed that into a nonexistent problem. Standards might provide a solution to the API problem, too.

    Anybody remember SoftSwitch? Didn’t think so. And if we’re lucky, nobody will remember Gnip, either.

  • Great post Jim! Great discussion/comments ensuing as well.

    While we didn’t set out explicitly to build a “web esb,” enterprise needs/learnings/products definitely played a role in our thinking.

    The hundreds of web service API endpoints that have cropped up over the years have been great for everyone, however, they’ve caused incredible discontinuity in the system. From format and protocol differences, to calling methodology differences (polling vs. event driven systems), everything’s a scattered mess. While one day we hope to have solved the _complete_ matrix of these issues, we’ve started small to, at a minimum, raise awareness of these challenges, and to provide a solution.

    We certainly considered using out-of-the-box ESBs for our implementation, however, we (at the moment) currently don’t need most of the industrial strength that comes with most implementations, nor are we prepared to pay for such products. As Jim noted in his comment, we played w/ Queuing in our initial implementation, but migrated on for a few different reasons. It is conceivable to me that we return to those roots at some point in the future (for obvious reasons).

    In a year or so, I firmly believe we’ll be considered something of a “transformation bus” for the web at large. I believe we’ll morph to look more like traditional ESBs, and I hope that traditional ESB plays will learn from us as well.

  • One of the other problems Gnip is trying to solve is that of latency. API providers typically limit the number of queries that someone can make (often, it boils down to one query per second).

    This means that once a service is calling for 86,400 people, you can only run through the list once a day. This creates an ever-worsening latency problem. Gnip is trying to change APIs from pull to push.

    There are certainly a number of competing protocols (XMPP and Atom spring to mind) to do the same thing, but both require widespread adoption to become truly useful (as does Gnip).

    By acting as a protocol bridge, we can enable real-time delivery of user data around the web via a wide number of formats while we all wait anxiously for the day that a single push protocol become standard.

  • Rob

    The ever growing number of API’s could be a real problem, but I think David hits the nail on the head..

    “First, it’s never really an n^2 problem in the enterprise, because most systems need to talk to only one or two other systems — there are usually only a few systems that everyone needs to talk to.”

    ..and I think the same is the case for the web?

  • Jim Stogdill

    @ Rob – I guess maybe people have different experiences in different environments, and I’m certainly no ESB evangelist, but I’ve worked in financial services, B2B, and now defense command and control and I have seen ample evidence of point-to-point interface proliferation. I don’t know if I’d call it N^2 since it is rarely an everything-connects-to-everything problem, bit it’s definitely greater than linear. Whether it is mathematically N^2 or not is sort of moot though, the real issue is that point-to-point interface proliferation leads to a costly to change brittle enterprise, and I’ve definitely experienced that.

  • Squid is my ESB.

    GNIP is doing protocol translation, which can be a very leaky abstraction.

    I think the whole concept smells. Even if you handle the protocol aspect, every application supports different data elements. You are still going to have do point to point integration, it now just has another layer of interference to go through. You can’t solve this with an “enterprise data model”, because you don’t write every application to deal with every enterprise data element. All of the same point to point problems still exist, because the application has to be written to the data elements it needs.

    In short, the point to point problem is overstated, and gnip can be looked at as a protocol switching service that enables point to point integrations. It’s very nice for that. But the whole n^2 is nothing to do with that. It’s really only solving the protocol to protocol problem- so it’s the number of channels you use squared, not the number of systems.

  • you may want to look at vtd-xml, the latest and most advanced xml processing api that is a must have for high perfomrance ESB


  • Jeljer

    Hi Jim!

    Nice post! You just dazzled me with this article!
    It so happens that I am a graduate student in the Netherlands and I just started with my final thesis (researching the possibility of using a ESB for a CMS) .

    A few months ago I had to write an article about the Enterprise Service Bus (click).
    I was very intrigued about the ESB and started to think about problems it could solve. After a few weeks of thinking about it on and off I suddenly realized it wasn’t only useful in trading applications etc. but also for web-hosting!. At that time i thought i had a unique idea. But apparently that doesn’t seem to be the case…

    But to make a long story a bit less long. I just begun researching the integration possibilities of an ESB for a Content Management System.

    I also read the comments above me (especially the negative one’s). I think the ESB will help with the hassle of all those different API’s, web-services and protocols. The list of things websites are able to do these days is huge! (googleDocs, googleCalander, Twitter, Facebook, Mail support, Flicker, Youtube, local new sites, etc.). However not only will the ESB help enterprises with the burden of integration but also will increase stability. If one web-service fails the whole system doesn’t have to!

    Greets Jeljer