John Willis [10.23.07 01:34 PM]

There are some really interesting companies working around S3/EC2. Now you add a little puppet with elastic computing cloud with a touch of serivices now you got yourself a VC play. Opps, I forgot VC's don't like services...

johnmwillis.com

  Luke Kanies [10.23.07 02:34 PM]

This article isn't clear on it, but iLike actually used Puppet to scale with all of those new machines:

http://blog.hjksolutions.com/articles/2007/08/31/puppet-ilike-and-infrastructure-2-0

(I'm the author of Puppet, but iLike isn't a client of mine and I'm not that author of that post.)

  John Allspaw [10.23.07 05:42 PM]

IMHO, bringing up ops considerations early in the process of product design should be a priority. Having at least some awareness of operational constraints can pay off significantly later on.

I might even add that while not being able to acquire hardware fast enough is by far a better problem, streamlining your procurement process should be also considered part of the whole system. We've had to learn a bit of that at Flickr, just like iLike. :)

As usual: great post, Jesse.

  Steve Loughran [10.24.07 08:39 AM]

-deployment should be part of every project, be it startup or in-house. all too often, the Enterprise specs (say, EJB) make assumptions that you have some process that involves people editing XML files in a GUI as part of the release process -the reality is that nobody wants to waste time on the dull bits.

-if you can bring up a system and deploy to it during the build, then you get to do functional testing on it.

-if your apps diagnostics are in a form that the ops team can use and understand, then everyone benefits.

I think where startups are special is there is less of a barrier between ops and dev; ideally: none. There is also the need to ramp up capacity based on demand, without the money to invest in custom datacentres. Outsourcing your servers to server farms is the sensible action here -and once you do that, the value of automating deployment really pays off.

But again, what's to stop the enterprises doing that, other than an urge to waste money and people on a problem that has already been solved, and a fear of taking risks.

Steve Loughran, SmartFrog team.

  Peter Nickolov [10.24.07 04:51 PM]

While I agree with Jesse that traditionally operations have been a large part of the secret sauce, I think this demonstrates how broken the current "traditional" operations are.

A software startup should be able to focus on (a) the main/unique value they provide -- their application's primary functionality and (b) how to sell it better.

Operations is a price of entry and should *not* be a differentiator. When each bright new startup needs to figure out the same things over and over again, this brings friction in the process. We as an industry have already figured how to do operations, it shouldn't be hard. It should be productized, as many other former differentiators are (remember the times when each WYSIWYG application had its own GUI system? each content publisher had their own markup language?). The Googles of this world may tell you that you really need to put a lot of money and innovation to build your own datacenters, your own computers from motherboards, etc.... You don't.

A second note: anyone who needs to scale (read: any Web 2.0 company with ambitions, which means, any Web 2.0 company) should carefully look at hosted options. Hosting providers have lots of servers at hand and can deploy 100s of them in a few hours. The iLike story would have been all about how easy it was to scale rather than how they had to beg and borrow servers.

Disclosure: I am the founder of 3Tera, the maker of the AppLogic grid operating system for web applications. Our best partners are hosting providers who offer virtual private datacenters (e.g., The Gridlayer).

Whether you choose our solution or do something different:

- make sure you have the ability to focus on your primary functionality and unique advantage

- ensure that operations are taken care of in the most outsourced way possible

Note to VCs: how long are you going to have the same 1MM or so spent in each startup for the same thing, namely operations? Find a way to not reinvent the wheel for each company (or at least not reinvent the wheels that they can afford not to reinvent

Best,

  Nelz [10.26.07 10:29 AM]

To address Steve's point about diagnostics:
As a software engineer / developer, I found Michael Nygard's book "Release It!: Design and Deploy Production-Ready Software" really illuminating and supportive of your point. Since reading that book, diagnostics and run-time reporting have become some of my battle-cries. (I'm just challenged by getting the "product owners" to see the value.)


Again, this is from a programming perspective, to effectively scale your application consider avoiding session-state. There are ways to program around this concept. (If you'd like to read an aggressively-titled blog entry on this concept, see http://davidvancouvering.blogspot.com/2007/09/session-state-is-evil.html)

  Brian [10.27.07 08:01 PM]

This advantage is the ability to consistently create and deploy reliable software to an unreliable platform that scales horizontally.

I've been in 'operations' most of my career; it suits me. How nice to see that other people get how important that area is.

One problem I've faced is that Operations is easy to ignore; most orgs can't use it to generate revenue, it's not as 'sexy' as development - on a good day in Operations nothing exciting happens, which can lead some to think that whatever money you're spending on that group is easily cut.

  Brian [10.27.07 10:39 PM]

But again, what's to stop the enterprises doing that, other than an urge to waste money and people on a problem that has already been solved, and a fear of taking risks.

Define 'Enterprise'? My day job is a mid-sized company, 8,000 employees worldwide. If you define 'enterprise' computing as 24/7 365 we're it.

I am not privy to all of the negotiations but we have looked at this ... and we've not found a host that suits our needs and can beat the costs we incur for maintaining our own data centers.

Outsourcing what makes sense - I"m for it. But for us, at least, it doesn't make sense.

  WordPress SEO [11.10.07 12:07 PM]

I am going to play devil's advocate and say that in the case of the startup getting the prototype up and running is more important. The application can always be streamlined later, more programming muscle applied, more hardware.

Moreover, having lead programmers involved in operations as you go means that they will know where the sticking points are.

And I agree with Peter that you want to outsource hosting to somebody dedicated to that space. Since we moved our clients to a first class host (Cartika), life has gotten a whole lot simpler.

When we are ready to scale they have a grid in place. We won't have to think about that.

  Stas [11.15.07 03:00 PM]

Hi.

Just wondering what in the end you've selected: Hyperic, Zenoss, or Groundwork?

  Brody [03.19.08 01:30 PM]

Maybe they need better servers or at least something scalable. Have you guys checked into Sun Microsystem's startup essentials program? Looks like a sweet deal.