Web Development 2.0

I gave a rushed and somewhat incoherent talk at the Y Combinator Winter Founder’s Program last night (and let me say again — holy cow, they have such great taste in people — the companies they’re funding are filled with people I’d love to work with). In it, I reprised a part of the talk I gave in Israel, cataloging some of the software development practices I keep running into at Web 2.0 startups. The interesting thing to me is that so many startups and companies seem to be developing a new set of software development practices independently, and only after the fact sharing them with each other. Software isn’t written for Web 2.0 companies the way it was during the bubble, nor is it written the way traditional, shipped software was. New ideas about Web applications seem to necessitate new ways of making those applications. Below is my current catalog of observations (less rushed than last night, and hopefully more coherent); I’d love to have more such practices suggested in the comments. How do you make a Web 2.0 app, and what kind of person is great at it?

  • The shadow app: I often hear developers say that their job is not to develop one application, but instead to develop two apps — the public-facing application, and the private application, the “shadow app,” which helps the company understand how the first application is working. Of course, statistics packages and traffic monitors are as old as the web, but these companies are explicitly rejecting any standard, pre-packaged code for this purpose, and are instead asking the questions they need for their specific businesses. One example: Flickr had a report of users with no contacts in the Flickr social network, which they called the “Loneliest Users” report. What a great report — a way to see who is uploading photos but not sharing them with anyone! With that, they could go add themselves as contacts for these “loneliest” users, and teach them how to use the contact feature. HitBox isn’t going to give you that report. The direct connection to your users provided by a server-hosted web app only gives you more data if you know what questions to ask, and building those questions is often just as important as building the public app iteself.
  • Sampling and testing: With tens of thousands of site visitors a day, or many more than that, the entire structure of engineering discussions has shifted heavily into the realm of statistics and controlled experimentation. Is this feature a good idea? Let’s show it to 0.1% of our visitors today, and see how they react. Is option A better than option B? Let’s try them both with 10,000 users each, and see which one works better. Why argue when you can find out the right answer from the people who matter, your customers? Feature selection feedback loops used to take months or even longer — and usually, the arguments about the right decisions were made nearly in the dark. The best software organizations made decisions based only on what customers said they wanted, which is often much different from how those same customers really act when presented with a new feature. With live sampling and testing, developers can see how many clicks the new feature really got — the impulses of the animal mind — not just how many people surveryed responded to the idea of a possible feature — the conversational bias. Some developers have complained about lost work from this approach — implementing the same feature four or five different times before it makes it live to the site. But most of them will admit that the feature is better in the end. Usability testing with a video camera over the shoulder and a one-way mirror in the room is giving way to usability testing through data analysis from a real, but small, deployment. None of the companies in this set have done over-the-shoulder testing at all, except for informal hallway tests.
  • Build on your own API: Of course many web app startups provide APIs, so external developers can build apps on top of their functionality and data. I was surprised to hear, though, how many of these companies build their own public-facing web sites second, by building on top of a web services API they develop first. The act of developing a public API, then, is not one of designing and testing various API calls — instead, all they have to decide is which of their existing method calls they want to expose to the public. They already know the methods work, because if they didn’t, the public web app wouldn’t work, either. In addition to “pre-testing” the API release, this also allows a very clean separation of responsibilities. One developer or set of developers works on the application’s “kernel,” exposed through the API; another works on the “view” the company exposes through its web site. In the operating system world, this is exactly the same as the separation between the Windows team and the Office team at Microsoft. It’s interesting, and very encouraging, though, to see the same model appearing at startup after startup.
  • Ship timestamps, not versions: Gone are the days of 1.0, 1.1, and 1.3.17b6. They have been replaced by the ‘20060210-1808:32 push’. For nearly all of these companies, a version number above 1.0 just isn’t meaningful any more. If you are making revisions to your site and pushing them live, then doing it again a half hour later, what does a version number really mean? At several companies I’ve met, the developers were unsure how they would recreate the state of the application as it was a week ago — and they were unsure why that even matters. A version number of an application is very convenient if you and your customer need to agree on the bits you’re each examining while searching for a bug or suggesting a feature. But would you really roll del.icio.us back to the way it was when a bug report came in, just to verify the report? Traditional QA would tell you yes (but see the next bullet); these developers just can’t see the point. If the bug can’t be reproduced against the live site, then what the hell does it matter? Other developers have a somewhat more formalized process — one company I visited added a label to source control (in the format I used above) every time the application was pushed to the live site. How many labels are on the source tree? I asked. “About 3200.” For them, too, the version number is dead. (And yes, I certainly see the irony of the term ‘Web 2.0′ in this light…)
  • Developers — and users — do the quality assurance: More and more startups seem to be explicitly opting out of formalized quality assurance (QA) practices and departments. Rather than developers getting a bundle of features to a completed and integrated point, and handing them off to another group professionally adept at breaking those features, each developer is assigned to maintain their own features and respond to bug reports from users or other developers or employees. More than half of the companies I’m thinking of were perfectly fine with nearly all of the bug reports coming from customers. “If a customer doesn’t see a problem, who am I to say the problem needs to be fixed?” one developer asked me. I responded, what if you see a problem that will lead to data corruption down the line? “Sure,” he said, “but that doesn’t happen. Either we get the report from a customer that data was lost, and we go get it off of a backup, or we don’t worry about it.” Some of these companies very likely are avoiding QA as a budget restraint measure — they may turn to formal QA as they get larger. Others, though, are assertively and philosophically opposed. If the developer has a safety net of QA, one manager said, they’ll be less cautious. Tell them that net is gone, he said, and you’ll focus their energies on doing the right thing from the start. Others have opted away from QA and towards very aggressive and automated unit testing — a sort of extreme-squared programming. But for all of them, the reports from customers matter more than anything an employee would ever find.
  • Developers — and executives — do the support: This one came as no surprise to me — I’ve often found that the best way to motivate developers is to let them see just one flamemail deriding the bugs in their work. What was encouraging, though, was to see how deeply that has made it into the development culture in these companies. Of course, this is a savings measure, too — but many people now seem to have decided that the best way to focus the efforts of developers is to make them respond to complaints directly. One company I saw made developers write the first five responses on a problem, after which the responses were edited by a support staffer and added to the canned response set. More often, though, the developers and the CEO respond to the majority of the support email. One CEO told me he responds to about 80% of all the mail they receive. How better to know what people are saying about your product? he asked. That seems like an unusual case, but still more common than a company with a support staff, which I didn’t see at all.
  • The eternal beta: This one is the most obvious, and the best-discussed. Following Google’s lead, many companies stick “beta” on their logos and leave it there for months or years. Gone are the betas that get released to a limited set of known but external testers, with formal product management follow-up interviews. The concept of “beta” as a time period or stage of development has fallen away, and been replaced with beta as a way of setting expectations, or excusing faults, about the current state of the application. (Not everyone agrees; Jason Fried and the crew at 37signals are noted contrarians.) I’d argue, though, that this is just the externally-visible artifact of all of the practices I’ve listed above. If you’re going to rely on customer reports for QA; if you’re going to change the operation of the app, however subtly, multiple times a day; if you’re going to introduce features to a small set of users, and then take them away at the end of the day — the experience your users will have is fundamentally different. “Beta” is one way of alerting them to the new regime. My question, though, would be, when would you ever remove a “beta” label if that’s really what it means? Does the “beta” come off when development stops altogether? Or at some point, do you call it “5.0” and put it into maintenance mode?

There’s a lot to say about these practices. A different kind of developer is needed to meet all of these needs; “beta” really seems to mean “2.0,” so maybe some other term would be better than beta; in the rejection of the old ways, what are we gaining and what are we losing? But for now, I’ll just start the catalog and say — I’m impressed with these teams and the way they work. “Continuous integration,” a phrase from the XP world, seems insufficient to describe this kind of software development. The app needs more than to build and pass tests at any time — it needs to be able to go live at a moment’s notice. For some that might be scary, but I think the result is a much more direct and honest connection with the desires of the user. In some ways, this is a realization of the goals many developers had when first coming to the web ten years ago — as last night’s host, Paul Graham, has said, “the web as it was meant to be.”

UPDATE: I added a follow-up covering some of the interesting responses I’ve gotten to this article, particularly around the “developers do QA” section.

  • Murray

    You’re piece reminds me of the web cheerleader articles from the last web bubble.

    There is a mix of good and bad practices here.

    Good
    1. the shadow ap – This is just writing queries to analyse user behaviour which everyone with any sense is doing.

    2. Sampling and testing – a great new practice that comes from the direct marketing industry – not many do this

    3. Build on your own API – comes naturally with modern tools such as XML and OO design

    4. Ship timestamps not versions – Duh! This is inevitable for a web company.

    Bad

    5. Developers — and users — do the quality assurance – In my experience 40-50% of code by web developers in an agile environment fails it’s first formal system test. Do you really want to inflict this on users? I think not. This is not a new practice. It’s a very bad, old fashioned practice that immature development groups are always falling into. Test driven development is a million times better.

    6. Developers and executives do the support – Yes exec’s and develoeprs should know and care about what customers think but most support queries are basic questions that can be handled by a customer service person. If they cant handle it they can pass it on to a developer or exec. Exec’s can always get a regular briefing on customer feedback. This waste of expensive resources can only happen in a small company that cant afford customer support.

    7. The eternal beta. – Trialing new sites with a limited user base is a good way to find out what people like and dont like about it but sending a beta out to everyone is just an excuse for delivering poor quality

  • http://radar.oreilly.com/marc/ Marc Hedlund

    Murray,

    Thanks, I appreciate it! I’ve been trying to be less cynical in my blog posts. :)

  • Leon Galindo

    This post and the corresponding comment accurately reflect the exact issues our company is tackling as well as the way we are coding.

    I share Marcs’ enthusiasm about the 2.0 Way and appreciate Murrays’ comments because they are correct — exec’s, developers and customers must be cautious in being too gung-ho about real-time developing, enabling & QA.

  • grumpY!

    the “beta” moniker is pointless and annoying. i would rather see a “changeInfo.html” page on a site that tells me when the last code change was and how many errors have been reported and fixed since then. we all know the codebase is always changing – i want to know its measure of safety. in any case the beta tag hasn’t done much for web app developers…if you put up a site and it sucks or is broken, users will not come back, be it alpha, beta, or “gold”.

  • http://alexanders.typepad.com/thinksum/ Jonathan Alexander

    Very interesting write-up, but yikes to those companies who espouse no QA team. Sure, having no QA eases the budget and reduces the time it takes to “release” (both big issues in start-ups) but Web 2.0 doesn’t reduce the usefulness of formal QA. It’s not revolutionary to under-invest in QA, it’s just a poor business decision that lots of start-ups have made for a long time, long before Web 2.0. I posted an entry on this here.

  • http://blogs.zdnet.com/Hinchcliffe Dion Hinchcliffe

    Marc,

    What a terrific write-up on Web 2.0 best practices. You capture the spirit of many of these beautifully. I tend to agree with some of the other commentors that letting your users do quality assurance is a risky practice in most environments. While it probably works for passion-driven companies, it doesn’t seem to translate well to larger organizations.

    Also, while what you list above in your catalog is primarily a development view, the one bit you really missed was using lightweight development tools that fundamental encourage change, improvement, play, and blendability. Yes, I’m talking about dynamic languages and frameeworks like Ruby on Rails, PHP, etc. These really seem to encourage the mindset of the continuous release and hackability while making it easy to fix bugs.

    Keep up the great observations.

    Dion Hinchcliffe

    Editor-in-Chief – Web 2.0 Journal

    ZDNet’s Enterprise Web 2.0

  • Perrin Harkins

    If “developers were unsure how they would recreate the state of the application as it was a week ago”, they simply don’t know how to use version control. It’s one thing to do frequent releases. It’s a very different thing to fail to track them in a way that allows releases to be reporoduced. Besides obvious issues like how to create a valid test system, or how to work on features that take too long to be done in the main code branch, this kills the developer’s safety net. This kind of thing leads developers to comment out their code instead of deleting it, or leave dead files around “just in case.” Effective use of a version control system solves all of these problems.

    Also — having no place except the live system to try to reproduce reported bugs? Who could possibly think that’s okay? What if the reported bug has serious side effects to the application or users? Bad idea.

  • http://radar.oreilly.com/marc/ Marc Hedlund

    Perrin, I think that’s a simplistic view. Many of these applications are based entirely on the contributions of users; what is, say del.icio.us with no bookmarks in it? Of course you can roll back the code to the last checkin before a push (note the label system I describe in ‘timestamps’ section), but rolling back the database to the state it was in at the time of the last push, or the time of the bug report, usually isn’t possible (unless the bug coincides in time with a database backup).

    The state of the code is preserved by source control. The state of the database isn’t, and is just as important in nearly all of these apps.

  • Perrin Harkins

    A responsible development team will have a test suite that includes faking the data required to exercise all the features of the application. Rolling back the live database should not be needed. Sure, there are some bugs that are timing dependent or only happen with very specific data, and are hell to reproduce, but those are not the common case. Saying that you won’t look into a bug unless you can reproduce it on the current live site sounds both reckless and lazy. In short, very web 1.0.

    Having lived through the original boom and still being at it, my opinion is that automated testing is by far the biggest change I’ve seen in modern web development. This was not common at all in the old days, but now it’s becoming a part of the common culture, and across multiple languages as well. It’s definitely a change for the better.

  • http://www.davewsmith.com/ Dave Smith

    “Tell them that net is gone, and you’ll focus their energies on doing the right thing from the start” is a believable statement coming from a manager whose team is doing test-driven development (TDD) without QA backup. The thought of having no QA backup used to scare the daylights out of me. Then I had the experience of working with a highly disciplined XP/TDD team that had minimal (but good) QA support. High code coverage through a large test suite, plus the discipline of requiring a clean test run before new code is checked in, can give an organization a lot of confidence. It also helps make any defects that do sneak through easy to find and repair.

    That said, waiting until a customer sees a data-loss problem that’s been observed in-house is highly dubious practice. Letting customers set development priorities can be a good practice, but waiting for them to complain that your app just trashed their data is pushing that idea a bit far.

  • http://radar.oreilly.com/marc/ Marc Hedlund

    Dave,

    I agree — see the follow-up I posted right as you were writing this.

  • http://schneider.blogspot.com jeff schneider

    Nice article – but I would consider modifying your rules on versioning. IMHO, the purpose of the version is to:

    1. identify changes AND

    2. identify changes that break backward compatibility.

    If a version is marked XX.YY.ZZ, changes in either XX or YY are said to break backward compatibility, while changes in ZZ are not. In addition, I would suggest that you are no longer versioning an application, but rather versioning multiple services, where each service maintains its own version.

    As you stated, versioning an entire site or application is almost worthless. The granularity of the entity being versioned must become more granular.

    Just something to noodle on – – Jeff

  • Todd

    Sir you are completely out of touch with reality.

    Rather than placate your editor’s demand to write about “Web 2.0″, you should do your research.

  • http://www.andyhoppe.com/ Andy Hoppe

    Very good points, very well formulated. One point I see with some concern: The developer interacting with the user will need to have a certain set of interpersonal skills, language skills etc. in order to avoid embarrassment to the employer.

    The IT world may thus become just another sector where every worker must come with a wide spectrum of competencies, at the cost of very specialized skills. Most really fantastic developers were rather shy folks who had none of these other qualities, but just did the one thing they were great at.

  • http://www.alierra-software.com Helen, software developer

    Andy the best way is to divide functions.
    Usually there are some people responsible for communication with clients and other who possess specialized skills.

  • Joel Maslyn, QA Dude

    Interesting, I came here to get some ideas on how to test web 2.0 as we integrate it into our application and find out we don’t need QA. Can I forget about the schedule now and take a much needed vacation?

  • Laboremus

    Man, this doesn’t make any sense. Whatever you are talking about will not work with 10,000 customers, who pay real money. It will perfectly work with 10,000 visitors who don’t really care.
    Developers responding customer? Ridiculous! CEO overseeing bug fixing? No, thanks!
    Start checking companies with more than 5 employes, more than 1 self-contained product and more than 5 customers. Get real!
    And another thing – Web 2.0 is realy a buzz word.

  • Dean

    The people concerned about version control for the sake of “backward compatibility” or failure “to track [releases] in a way that allows releases to be reproduced” seem to be missing the point. We are talking about web applications, not desktop apps. There is only ever _one_ version of a web app in release — the live version. If there was a bug two releases ago and it’s not in the live version, who cares? And does “backward compatibility” have any meaning in a web app, like del.icio.us, flickr or Google? No user is running any other version but the current version.

    Version control is a crucial development tool even in a web app, but not for these reasons.

  • http://gailwilliams.wordpress.com Gail Williams

    Interesting! Glad to have stumbled on this. One thought from watching support and developers for many years:

    Quite a few brilliant coders seem to have that “never suffer fools” approach to customer service. It’s not a personality disorder in a developer, but it’s toxic in a support staffer! You’d hate to have to hire programmers with an eye to their ability to soothe ruffled feathers and assist the clueless.

  • http://www.myuninstalledlife.com/top-10-reasons-why-web-20-apps-will-work-in-10-years Jonas

    I’ve written an article why web 2.0 applications WILL work – but it could take some 10 years. Let me know what you think. On myuninstalledlife.com I’m trying to uninstall everything from my desktop and replace it with web applications – and I mean EVERYTHING, I just don’t want to replace the apps but also keep 100s of gigs of data online.

  • http://www.speedtracker.de Speedy

    If “developers were unsure how they would recreate the state of the application as it was a week ago”, they simply don’t know how to use version control. It’s one thing to do frequent releases. It’s a very different thing to fail to track them in a way that allows releases to be reporoduced. Besides obvious issues like how to create a valid test system, or how to work on features that take too long to be done in the main code branch, this kills the developer’s safety net. This kind of thing leads developers to comment out their code instead of deleting it, or leave dead files around “just in case.” Effective use of a version control system solves all of these problems.

    Also — having no place except the live system to try to reproduce reported bugs? Who could possibly think that’s okay? What if the reported bug has serious side effects to the application or users? Bad idea.

  • j

    “If developers were unsure how they would recreate the state of the application as it was a week ago, they simply don’t know how to use version control.”

    It’s a website–why would you ever want to go back and look for a bug in the old site if it doesn’t exist in the current one? The old one’s already gone. Vertsion control is pointless here, just a waste of space.

  • http://www.l8r.nu/ joost baaij

    You’ve written one of the most complete and sensible descriptions of 2.0 I’ve read so far. And I agree with most of them–especially “Ship timestamps”.

    Version numbers have become irrelevant, and that’s exactly what I see with my own app. Who cares about version x when you’re able to roll out a bugfix within minutes? (Capistrano is magic in this regard)

    I just don’t agree with the ‘beta’ moniker. Either stuff can be used and you call it a release, or stuff’s not safe for consumption yet and you call it a beta. When you’re letting people in, don’t muddle the waters with slapping a beta badge on your product. Real Men(tm) ship finished products!

  • http://www.wrestling-radio.com Timothy Armstrong

    I tend to believe QA on the users is unnecessary at large but kind of the “way it goes” whether you like it or not, the live code, tried and tested can’t always be reliable in that off situation the most common user breaks it.

    However, I find the most live bugs in a project come from those who plan poorly, the client, the developer, someone in the line op production.

    Often the pressure to go live vs. the need for it being “done right” is often the client vs. developer battle as well as the clients needs for you to revise it too much, due to this lack of planning.

    Good article though, I definitely agree about web versions being conventionally 1.0 .. 2.0 are done, especially when a feature may react one way, 100 or so users already hit it, then the developer wacks it on the head or changes its behavior 10 minutes later.

  • http://www.broadwayinfotech.com Web development

    Web”>Web”>http://www.broadwayinfotech.com/web_development/web_development.shtml”>Web development is not just about designing an eye-catching website. It incorporates activities partaking to successful integration of security and serviceability with a website with aesthetic look and feel.

  • http://www.cyberdesignz.com/ jhon

    Great article the web development 2.0 now are used very largely,http://www.cyberdesignz.com/ ,A responsible development team will have a test suite that includes faking the data required to exercise all the features of the application .

  • http://www.go-gulf.com Karel Zeman

    I have work on most of application, especially on desktop application and also on web applications. My philosophy is if application is easy, user-friendly it will help the user to understand quickly. Most importantly not with errors.