Sections

• A return to first principles
• HTML5: Still connecting things
• HTML5 connections are the new rich media
• JavaScript isn’t the focus of HTML5 … right?
• Container-based web pages: A step (sort of) in the right direction
• The canvas element is a programmable div
• Mobile: Killer application, ho-hum client
• … and a partridge and a pear tree

Download this report for free

HTML5: Everyone’s using it, nobody knows what it is. I realize that sounds more like a line out of an existential movie — maybe Waiting for Godot or a screenplay by Sartre — than a statement about HTML5. But it’s really the truth: most of the people using HTML5 are treating it as HTML4+, or even worse, HTML4 (and some stuff they don’t use). The result? A real delay in the paradigm shift that HTML5 is almost certain to bring. It’s certainly not time to look away, because by the time you look back, you may have missed something really important: a subtle but important transition centered around HTML5.

In this post, I want to take a deeper look at HTML5. I have a simple proposition with a lot of complex consequences: HTML5 is both something entirely new, and yet nothing more than HTML was ever intended to be; and that once you really understand HTML5, you’ll change the way you code and even think about the web and your own web applications.

A return to first principles

HTML has always been about interconnection. Back in the ancient days, when electronica was cool and not called “house music” and before the Rolling Stones qualified for Medicare, the web was littered with big huge documents. In fact, it was exactly the opposite of today, where most people think enhanced digital books are just electronic wrappers around full-text copies of what’s in print.

In the ’90s, the web was full of 15-page specifications, all in a single file. You scrolled through those massive documents just like you paged through an encyclopedia. Much of the early versions of HTML were intended to deal with this, widely considered a detriment to the readability and usability of the web. That’s largely because Tim Berners-Lee, the recognized father of HTML, was a researcher enabling other researchers (mostly at CERN at the time). If you’ve ever known anyone mired in research, brevity is rarely their prevailing trait, so reading huge documents online was a necessity, but scrolling through 15- (or 1,500-) page documents just wasn’t a long-term option.

So early on, HTML was not primarily about displaying those documents with lots of formatting. Most fundamental to HTML was the simple a tag. It gave a document the means to link to another document. Suddenly 15-page documents were reduced (mercifully) to 15 one-page documents, all linked together. Bye bye scrolling; hello useful linking. This is all pretty standard fare, and if much of this is new to you, then the waters are going to get deep quickly.

HTML5: Still connecting things

Fast forward to the present. Ultimately, this ability to connect things on the Internet is still the primary feature of HTML5. It’s just that now, we’re starting to realize the original vision of HTML, and connect a lot more than hypertext with static images. So the introduction of the audio and video elements in HTML5 are nothing more than logical extensions of the old a element.

(Note: in a more correct sense, audio and video replace object and all the embed code that people have been throwing into web pages for years, largely pulled from sites like YouTube or Vimeo. Still, those elements are semantically functioning more like an a element that drops the link into a page, rather than taking you off to another destination. In that sense, even the img element is in some ways nothing more than an inline a element: it grabs content from another location and pulls that content into a page. It’s all just linking, and that’s what HTML is really about: linking and connecting things.)

So now you can pull in images, audio, and video directly into a document. More importantly, because those items now have first-order elements, you can easily manipulate the audio and video from JavaScript. That’s a big deal, and something I’ll come back to later. In a nutshell, a first-order element is always going to encourage more direct programmatic access than one in which you have to be sneaky or clever to really get at it.

So while the audio and video elements are new, their purpose isn’t. HTML5 allows you to integrate more assets into a single document, all while keeping the integrity and separation of those assets into place. This is nothing more than making sure that the bibliography of a document isn’t physically stuck at the end of a long web page, but is in fact separate and easily maintained, but still able to be integrated into the rest of the page.

Now, before moving on, you need to immediately see that it’s not (relatively) important that you can grab audio and video and drop it into an HTML page. What is important is that you can more easily grab “other stuff” and drop it into a page. There may eventually be 20 or 25 elements for things well beyond “audio” and “video,” and the fundamental premise is still the same: the important thing here is that multiple pieces in multiple places can all be wired together in a meaningful way, with semantic elements describing that “stuff” easily accessible by JavaScript. The fact that, in HTML5, that “other stuff” happens to be audio and video is cool, but rather incidental.

OSCON JavaScript and HTML5 Track — Discover the new power offered by HTML5, and understand JavaScript’s imminent colonization of server-side technology.

Save 20% on registration with the code OS11RAD

HTML5 connections are the new rich media

So what, then? Why is this such a big deal? Well, largely for three reasons:

1. Web pages no longer need to look (and act) like web pages. The rise of Flash over the last years has largely been an attempt to overcome “limitations” of what HTML allows. Flash was initially often focused on animations and cool visual effects. But then entire sites got rolled into Flash, allowing for different types of navigation and page organization, richer programmatic access to the individual pieces of a web page, and the ability to avoid the quirkiness of JavaScript. (I’ll leave out the obligatory comment here on the quirkiness of the Flash stack.)
2. Web pages no longer need to represent one person/organization’s content. Even though web programmers and designers have been pulling in images from other sites for years, web pages are still largely homogenous in terms of the asset ownership. A web page today really has one person’s content, images, pages, media, and the like. Even sites like Vimeo and YouTube are more often used as extensions of a private repository than an actual free medium for world access.
3. Web pages can function intelligently and easily across display devices. It’s no secret that MOBILE (as one co-worker recently email-shouted at me) is the banner under which HTML5 most often flies. But the story really isn’t that HTML5 has great mobile support; rather, it’s that mobile is no longer a problem child. In other words, the story is that what works on a desktop browser pretty much works on a phone. (The list of things not covered by “pretty much” is shrinking every few weeks, so better to not list them here and appear outdated next month.) Put another way, phones and tablets are first-class citizens, because they are privy to the same interconnections listed above. In fact, it’s probably not too meta-physical to say that in addition to inter-connecting content, HTML5 has a really good shot at interconnecting all the devices floating around … and that’s arguably at least as big a deal as what it does for content and web applications.

HTML5 introduces — although it’s not at all complete in its support for — a paradigm that moves away from both of these limitations. First, HTML5 and CSS3 provide for JavaScript a pretty solid working set of tools and effects, comparable to most Flash websites. I can generally take a medium-sized Flash website and recreate it in WordPress, HTML5, JavaScript (via jQuery), and CSS3 in a week, if not less. And the upside is enormous: text is again selectable, bookmarking works without lots of weird tricks, and of course website owners can actually update their own sites, rather than relying on some overly busy Flash programmer to help.

The result is that HTML5 is again the most usable and indexable tool available for web content; but that content is richer than ever before. Further, that content no longer has to be owned to the degree that older web pages had to be owned. For a lot of emotional and psychological reasons, mainly, the audio and video elements suggest pulling assets from other sources more than the more generic object element does. There’s something about seeing video that screams, “Grab some video and stick it in your page!” But since there will always be at least an order of magnitude greater number of web page creators than video makers, how do you get that video? You grab it from someone else, hopefully someone who’s used a nice Creative Commons license. The same is true for audio, of course: you can connect to someone else’s audio incredibly easily, and so you should. And as already intimated, the audio and video that plays on your desktop browser also plays pretty well on an HTML5-enabled phone or tablet.

Underneath these limitation-stripping observations is something much bigger: content creators are moving from creating completely original content to amalgamating and assembling content. Sure, this isn’t anything technically new, but it is something that’s happening more due to the introduction of HTML5 elements like audio and video. And when the web starts to grow in its interconnectedness, it takes a giant step toward the world of Neal Stephenson and “Snow Crash” and “Neuromancer,” doesn’t it? One person creates a video and doesn’t have to place it or build out a web page. They just throw it on a content-sharing medium like YouTube or Vimeo. Another person goes beyond a simple off-page link but actually integrates that video (fully accessible and mutable via JavaScript, something we’ll get to shortly) into a web page. Neither person has to be an expert outside of their own domain (video and web page, respectively), but the result is a fusion of assets.

Now expand the working set of that fusion by a factor of 1,000. Or 1,000,000. Throw in data science and the ability to find, organize, and even localize data like never before. Access that data not just from one browser, but all the major browsers, on all the major devices — laptops, netbooks, tablets, and phones. Toss in unbelievably cheap data storage, and then realize that we’ve yet to mention the increased power of JavaScript, the canvas tag and its ability to further manipulate the referenced audio and video, and web messaging, and you’ll quickly forget that we’re talking about HTML here, not a traditional high-end programming language like Java or PHP!

JavaScript isn’t the focus of HTML5 … right?

That greater ability of HTML5 pages to do a lot more brings JavaScript into focus. Honestly, there’s not much in HTML5 that specifically speaks to JavaScript. Yes, the HTML5 draft, according to the W3C, is intended to replace the JavaScript APIs detailed in older HTML4, XML1, and DOM Level 2 documents. But no, don’t expect the JavaScript language to be radically, or even that noticeably, changed in HTML5. Yet hang around the web programming water cooler, and you’ll hear more about JavaScript than ever before. So what gives?

Well, first of all, HTML5 adds several important new first-order elements, as already mentioned. audio and video are key, as is another element we’ll get to, canvas. That means that it no longer takes a lot of document.getElementsByTagName(“object”) or ID-tagging of object elements to work with media in a web page. Extending that, these elements (unlike the object element) have media-specific attributes like autoplay and preload. This means that with JavaScript, you can grab a video, display its controls, change the URL and effectively load a new video, and even control when the video loads (or preloads) and mute and unmute the audio.

All of this is done via JavaScript, but there’s nothing new in JavaScript enabling this access. It’s the addition of semantically meaningful elements in HTML5 that makes JavaScript more powerful, or at the least an easier medium in which to accomplish these tasks. Sure, a clever JavaScript programmer (who was comfortable tokenizing and parsing) could get most of this from grabbing particular attributes from an object element, adjust them, reset them, and so on. But my goodness, it was a mess, it was player-specific, and mostly made for annoying late nights trying to get the semicolon in just the right place.

In fact, much of what’s great, albeit seemingly boring, about HTML5 is the move of important elements and attributes from customized or one-off items to semantically important ones. Even the draggable attribute — allowing native drag-and-drop in HTML5 — is cooler because it’s now accessible and easily mutated via JavaScript. A page requires less textual data to mean something, and more of that meaning is pushed into the structure of the page itself.

I think it’s fair to go as far as being axiomatic here: the more a document has semantic meaning, and the less attributes or even elements are nested as plain text within hidden input elements or object embeds, the better that document will be. The page becomes easier to access, update, and make responsive from a JavaScript point of view; CSS (whether it’s CSS2 or CSS3) can be applied directly without so many messy pseudo- and nested selectors; and the document describes itself. And when you have documents that have semantic meaning, you’re going to find that your JavaScript is clearer. In fact, many mid-level programmers will realize that they can do things that the “advanced” programmers can: throw a video into the page based on a click, autoplay that video, mute the audio and load a different piece of audio, play that different audio over the video, and boom: you’ve got an all-HTML5 discussion by Hitler about JJ Abrams and Star Trek … without having to modify the original audio and video assets.

Put another way, the HTML page becomes more of a container (which contains other containers, which contain other containers … and so on) than a page. JavaScript can easily access those nested containers, and modify them, update them, move them around, and do anything else you can dream up and code. CSS can also visually style and work with those containers, further making the HTML a purely organizational tool, and even that becomes loose. With absolute positioning, the order and position of elements within the HTML organization becomes at best a guide, if not totally irrelevant.

(It’s worth saying a display approach that completely ignores the organization and sequence of elements within an HTML document is a pretty bad idea. What you gain with new semantically valuable elements is quickly lost when you start absolutely positioning everything, and removing any sense of hierarchy. Of course, that’s true in HTML4 and XHTML1 as well as HTML5, so it’s not something I’ll focus on beyond this rather quick note.)

Container-based web pages: A step (sort of) in the right direction

The idea of an HTML page being little more than an organization of malleable elements is nothing new, but HTML5 seems to be a clear jumping-in point for JavaScript programmers and web developers that haven’t yet made that conceptual leap. It’s notable that with the rise of CMSes like WordPress for smaller businesses and personal websites, almost everyone is now familiar with the separation of concerns involved when even the text of an HTML page isn’t necessarily encoded and stored within a static HTML file. This is all a good thing, too. As a JavaScript programmer myself, I’d rather have more programmatic control and create a page that is built based on user decisions, database-driven logic, and programming over one person with an HTML or text editor.

In fact, the most obvious example of this approach is the Twitter website. For example, when I visit Twitter, I see a ton of content:

But if you take a closer look, by viewing the source, you’ll find hundreds of lines of JavaScript at the beginning of the file, a ton more JavaScript at the end of the file, and this rather pedestrian bit of HTML squeezed into the middle. Here’s about 1/3 of that HTML, which is pretty brief:


<![CDATA[
<body class="user-style-twttr loading-body ">
<div id="doc">
<div id="top-stuff">
<div id="banners" style="clear:both;"></div>
<div id="top-bar-outer">
<div id="top-bar-bg"></div>
<div id="top-bar">
<div class="top-bar-inside">
<div class="static-links">
<div id="logo">
<a href="/">Twitter</a>
</div>
<form id="search-form" action="/search" method="GET">
<span class="glass"><em></em></span>
<input value="" placeholder="Search" name="q" id="search-query" type="text" />
</form>
<div id="global-nav">
<ul>
<li id="global-nav-home"><a href="/">Home</a></li>
<li id="global-nav-profile"><a href="/bdmclaughlin">Profile</a></li>
<li id="global-nav-messages"><a href="/messages">Messages</a></li>
<li id="global-nav-whotofollow"><a href="/#!/who_to_follow">Who To Follow</a></li>
</ul>
</div>
<div id="sections"></div>]]>


Not a lot to it, is there? Of course, even parts of this have been inserted programmatically, like links to my personal profile and messages. And even though this is an HTML4 site, it points to some important things with which HTML5 is attempting to deal.

Now, what I want to argue is that while this is a generally good thing — using JavaScript to connect the organization of HTML to content not encoded directly into that HTML — the step forward of new HTML5 elements is matched, if not overtaken, by some steps back in terms of this sort of programming, at least as it’s currently implemented. It’s not exaggeration to say that many pages, like Twitter’s page, are basically a bunch of empty divs with id attributes and not much else. It’s unclear whether Twitter’s page will evolve into a more HTML5-centric approach, but unless it also evolves its structure, there are some problems.

Here’s the thing: this approach is programmer-rich, yet semantically poor. First, elements aren’t truly indicating content; they’re just indicating buckets that can be filled. And don’t mistake a textual value attached to a div‘s id attribute as semantic meaning. That’s no different than the object element when used to display audio or video; you’ve got to parse and tokenize to get any real sense of meaning. That’s meaning, but it’s not meaning encoded into the HTML in a particularly useful way.

This is another area that HTML5 seeks to improve. For many, the introduction of another few elements — nav and header and footer, for example — have been largely ignored. Why not just keep using <div id=”nav”>? Well, now the answer should be obvious: these new elements provide additional semantic meaning. If I’m linking to your page, or even pulling in part of your page, in my own page, I can grab (or not grab) your footer, your navigation, your figures and figure captions.

Don’t let the SGML-ness of semantic meaning escape you. (Yes, I realize that SGML-ness isn’t a word, but it should be.) When you really begin to connect heterogeneous parts of the web together, semantic meaning becomes huge. So does a page that is more than a bunch of divs with id attributes. You’ll want to know what you’re getting, and an HTML standard is a heck of a lot better way to determine meaning than arbitrary text values tucked away into id attributes.

By now, you’re either seeing a theme in what I’m calling out about HTML5, or you’re seeing little more than a grab bag of loosely connected features. If you’re in the latter camp, let me make it explicit: HTML5, when used both as the 21st century web suggests and as the original HTML specification allowed, is best at interconnecting things. If you view your pages as a collection of content, and let go of the rather egotistical idea that all that content has to be your own, then all of the new features of HTML5 discussed so far are hugely important. You can pull in audio and video and manipulate that audio and video as if it were your own. You can organize your page semantically and link to and even pull in content from other sites, using the semantics of that page. You can separate content from organization using more than just divs, but actual first-order elements for navigation and headers and footers.

And then, when things were going so well, there was canvas.

The canvas element is a programmable div

It’s odd that the element that is probably at the top of the HTML5 spec in terms of “wow factor” is in fact the element that brings in the ability to undo most of the good I think HTML5 does. That element, of course, is canvas. canvas defines, well, a canvas on the HTML page. Using JavaScript, you can draw on that canvas using methods that look a lot like a graphics scripting language. JavaScript basically grabs the canvas by its id attribute, using document.getElementById(), and then gets a context. Most of the examples online that are really jaw-dropping use the 3D canvas, but there’s a 2D canvas that’s a lot better entry-point if you’re new to this sort of thing.

Once you’ve got a context, you can operate upon that object, using properties like fillStyle, and methods like fillRect(top, left, width, height);, and canvas. Yes, it’s really that simple, and even pretty intuitive. In fact, more than anything else, drawing on a canvas for the first time with JavaScript felt a lot like my old Logo days in grade school, or even Java AWT days in early Java. (Oddly, I’m prouder of what I did in Logo than AWT; don’t judge.)

I don’t think it’s worth a lot of ink to walk you through using canvases. There are tons of web tutorials out there in both 2D and 3D, and a slew of good video and publications. Suffice it to say that the days of needing Flash to really do anything visually interesting, or at least needing to take that visually-interesting thing you created and turn it into an image or a video, are gone. What’s even cooler is that you again get JavaScript integration into your page. So now a keypress or a form submission, or dragging an element on the page, or the playing of a video (because we have draggable and video now, remember?) can interact with the canvases on your page.

Now, there are two caveats: one small one and one giant one. The small one is that browsers are still getting their HTML5 act together, and the 3D context in particular is pushing browsers beyond what they were often intended to do. So realize that while you’re not going to get the dreaded “Flash content won’t play” sort of message on your iPhone, there is going to be a lot of rigorous testing required to get your 3D drawing just right on the most computers in the most browsers. But, honestly, that’s a testing issue, and one that both time (as people upgrade) and testing can overcome.

The more serious issue is that the canvas becomes much like an anonymous, programmatically-filled div. The things within it have no semantic meaning and existing apart from the HTML’s encoding on disk; it’s a dynamic set of programmatically-created lines and squares and spheres. No matter how cool of a texture you wrap around a 3D model, you can’t go in with a separate JavaScript function (or better, a separate HTML page) and grab that textured sphere, link to it, spin it around its axis, or adjust the light source. You can do all of those things from the page itself, mind you; you just can’t do it in the loosely-interconnected manner that is heralding the new and cool web that HTML5 helps drive.

So is this a stump speech against canvas? Of course not. I’d no more avoid canvas than I’d tell someone in HTML4 to avoid div. Yes, there is a limit to semantic meaning. Yes, the canvas becomes a big blob that contains something, but that something has minimal outside visibility. But it’s still a step forward, and heralds what is hopefully a wealth of GPU studs and studettes making HTML5 do far more than anyone thought possible. (Studettes: another made up word, but these are good words. Webster, anyone? Urban Dictionary?)

Hopefully, there will soon be standardized ways to call into objects that are created within a canvas, and granularity becomes part of the advantages of a canvas rather than a drawback. Until then, you’ve got to decide if using a canvas makes sense. In general, if you’re doing something page-specific, or if your canvas is itself a self-contained “mini-application,” then go forth and prosper. If your canvas is tightly integrated into the rest of your page, you’re probably in great shape, too. But if you want a lot of other pages and sites to pick up your canvas as any sort of component, you’re probably out of luck, and might want to look at a different approach.

The good/bad news here is of course, that HTML allows for this sort of reuse, and people are starting to figure that out, but people are just starting to figure this out. Interconnections again; are you building your own little kingdom in the web, complete with moat and drawbridge to keep everyone else out? Or are you integrating everything you do into a larger-than-you Internet that shares (and sometimes steals) as much as it creates, and in that sharing and stealing recreates?

Mobile: Killer application, ho-hum client

For those of you already read up on HTML5, you may be surprised how little the word “mobile” has shown up. Sure, I mentioned earlier that mobile devices can view HTML5 web pages, and consume audio and video from audio and video without many problems. But isn’t this the big story with HTML5? In fact, isn’t it mobile devices, more than anything else, that are driving HTML5 adoption?

Well, yes and no.

As far as mobile devices driving adoption, absolutely: HTML5 owes a lot of its buzz to mobile devices, and in particular Steve Jobs and his near-militant campaign against Flash. The iPhone came with Safari, Safari had HTML5 support, so suddenly iPhone users were giddy and talkative. Now almost all modern smartphones have an HTML5-capable browser. So while the ability to view video without Flash is big, the yet-bigger story is simply that anything that works on an HTML5 website as viewed within a desktop browser pretty much works on a phone. So yes, HTML5 wouldn’t be the talk of the developer town if not for mobile.

But no, mobile can’t be the big story with HTML5. (People will claim it is, and do so all the time, but they’re misguided in that they’re calling an effect a cause.) HTML5 doesn’t offer all sorts of bells and whistles for working with mobile devices. Yes, there are ways to detect the orientation and screen size of a mobile device; but those also allow for detection of the size of a browser window, or a tablet display. And yes, certainly, the canvas element and new audio and video features make it possible to deliver amazing content to a mobile device. And yes, yet again, more and more folks are building mobile-optimized websites for phones. So how is this not a big deal?

Well, what’s important is not that HTML5 works on phones. What’s important is that HTML5 “just works.” There aren’t tons of issues when getting your site to work on a phone … and that’s the story. The big deal isn’t that you can do all kinds of cool things with your website for mobile phones. The big deal is that you can do all kinds of cool things, period; and that those cool things just happen to work on mobile phones … and tablets … and netbooks … and desktop clients. HTML5 really removes the need to think about mobile devices separately from other devices.

(Note: I’m not saying you should ignore mobile devices, or not think about them separately. The best sites I use on my phone are almost all at least mobile-optimized, if not complete with a pure-mobile theme or CSS styling. That said, you don’t have to do this to get up and running. Also, almost all good mobile-themed sites have a “View normal site” that works perfectly well. That says a lot. It used to be you’d want to hide the normal site at almost all costs. Not with HTML5…)

… and a partridge and a pear tree

As this article turns the bend from readable to monolithic, at least in article terms, it’s easy to look at what else HTML5 offers, and consider writing an entire book. There’s web messaging, and offline applications, and local storage … really, everything you need to write an entire application. And ultimately, that’s the brief but accurate assessment I’d make of the “everything else” of HTML5. HTML5 is no longer about a single page, or even a collection of inter-linked pages. It’s about interconnection — and I’ve tried to beat that drum consistently above all else — but that interconnectedness is about a lot more than pages. It’s certainly about a lot more than just your pages.

Web messaging allows interconnecting HTML5 applications (I’ll defend the use of “application” here in a moment; for now, just go with me). Those applications don’t have to be in the same domain; for the first time, there’s an intelligent JavaScript-based messaging framework that allows for communication across domains. You can write listeners to receive messages and senders to send them. In its simplest form, all the limitations of Ajax sandboxing that everyone griped about have been removed, albeit with some security considerations and a lot more complex API than the pretty simple XmlHttpRequest object. But all the same, if you buy into the premise that HTML5 encourages sharing of resources across the web, then messaging makes that sharing more of a true two-way communication. Now you can build resources that can be consumed, even if that consumption involves receiving data as well as sending it.

And you’ve got local storage, too: a legitimate database-like storage mechanism for holding onto data and delivering it to a server on the Internet when that server is available — even if that’s not the “now” of the page’s existence and use by a real person. These tie in closely with forms, the most obvious means of getting this sort of data that local storage is optimized to store. In fact, while it’s nice that HTML5 forms offer some new controls and validation and the like, it’s their ability to interact intelligently with offline storage (via, again, JavaScript) that’s the really big story. Think about that: for the first time in, well, ever, there’s valuable offline access that can take place. Again, there are kinks (and some of them are big) and some new wrinkles to take into your JavaScript, but the web has always had kinks and wrinkles. It’s just that now you’re getting more than ever for dealing with those issues.

Add to that even heavier-duty types of functionality, like threads (which are called web workers, but really are just threads) and sockets. On the one hand, these aren’t part of the core HTML5 spec. Instead, they’ve been moved into WHATWG, so that says something about their ultimate inclusion and importance as part of HTML5 to the W3C community. On the other hand, by untying these from the core spec, they may now be free to evolve further and more quickly than ever before. In any case, it’s no small thing to be talking about threads and sockets not as part of code that generates or produces HTML, but as part of an HTML application itself. These are serious tools, and further indicate that HTML5 is not to be taken as a language for building interesting and largely static web pages anymore.

So about this word application: isn’t that what’s ultimately being created with HTML5? This isn’t just about neat visuals and clever drag-and-drop controls and even replacing Flash-based sites. HTML5 pushes a good programmer to create semantically-meaningful and accessible resources. Those resources don’t have to be entire pages. They can just be “things:” video and audio and navigation and footers. Those “things” can not only be used across domains by other “things,” but there’s actually communication that can happen. You can pull in a very well-designed footer by sending it a few choice bits of information, and have that footer pulled from one domain interact with a video pulled from another domain. When the user is offline, big deal. Local storage not only allows for database-like storage of an HTML page without an installed database, but is even kind enough to make sure that data gets to the right place where connectivity is restored. JavaScript is not a toy language anymore, and jQuery makes what was annoying in JavaScript lexically a piece of cake.

These are applications.

Stop building web pages. Please. Stop trying to only use “your” content. Slap a Creative Commons license not just on your video, or your images, but on your HTML, and then build interesting components and assume they’ll be sucked into other things and used in ways you’d never considered. It is called the Internet, remember?

The winners in an HTML5 world are those who stop fearing being stolen from, and actually start handing out their candy to every kid on the block. It’s about a lot more than YouTube-like sharing of videos. Applications are increasingly becoming a collection of mini-applications, and if you’re thinking about your HTML5 work as collecting resources — some original, some shared — then you’re way ahead of the curve.

OSCON 2011, coming up later this month, will have a track dedicated to JavaScript and HTML5. Learn more and save 20% on registration with the codeOS11RAD.

Related:

• What is Node.js?
• Can Flash and HTML5 get along?
• Why HTML5 is worth your time
• New directions in web architecture. Again.
• Where the semantic web stumbled, linked data will succeed

Sections

• A warning to the Node experts out there
• Node: A few basic examples
• Skipping hello world

Node.js. It’s the latest in a long line of “Are you cool enough to use me?” programming languages, APIs, and toolkits. In that sense, it lands squarely in the tradition of Rails, and Ajax, and Hadoop, and even to some degree iPhone programming and HTML5. Go to a big technical conference, and you’ll almost certainly find a few talks on Node.js, although most will fly far over the head of the common mortal programmer.

Dig a little deeper, and you’ll hear that Node.js (or, as it’s more briefly called by many, simply “Node”) is a server-side solution for JavaScript, and in particular, for receiving and responding to HTTP requests. If that doesn’t completely boggle your mind, by the time the conversation heats up with discussion of ports, sockets, and threads, you’ll tend to glaze over. Is this really JavaScript? In fact, why in the world would anyone want to run JavaScript outside of a browser, let alone the server?

The good news is that you’re hearing (and thinking) about the right things. Node really is concerned with network programming and server-side request/response processing. The bad news is that like Rails, Ajax, and Hadoop before it, there’s precious little clear information available. There will be, in time — as there now is for these other “cool” frameworks that have matured — but why wait for a book or tutorial when you might be able to use Node today, and dramatically improve the maintainability of your code and even the ease with which you bring on programmers?

A warning to the Node experts out there

Node is like most technologies that are new to the masses, but old hat to the experienced few: it’s opaque and weird to most but completely usable for a small group. The result is that if you’ve never worked with Node, you’re going to need to start with some pretty basic server-side scripts. Take your time making sure you know what’s going on, because while this is JavaScript, it’s not operating like the client-side JavaScript you’re used to. In fact, you’re going to have to twist your JavaScript brain around event loops and waiting and even a bit of network theory.

Unfortunately, this means that if you’ve been working and playing with Node for a year or two, much of this article is going to seem pedestrian and overly simplistic. You’ll look for things like using Node on the client, or heavy theory discussions on evented I/O and reactor patterns, and npm. The reality is that while that’s all interesting — and advances Node to some pretty epic status — it’s incomprehensible to someone just getting started out. Given that, maybe you should pass this piece on to your co-workers who don’t know Node, and then when they’re buying into Node’s usefulness, start to bring them along on the more advanced Node use cases.

Node Day at OSCON — This year’s OSCON features a day-long dive into Node on Tuesday, July 26. Join experts and users from the Node community to discuss best practices and future developments, and survey the ever-growing number of Node frameworks and plugins.

Save 20% on registration with the code OS11RAD

Node: A few basic examples

First things first: you need to realize that Node is intended to be used for running standalone JavaScript programs. This isn’t a file referenced by a piece of HTML and running in a browser. It’s a file sitting on a file system, executed by the Node program, running as what amounts to a daemon, listening on a particular port.

Skipping hello world

The classic example here is “Hello World,” detailed on the Node website. Almost everyone starts with Hello World, though, so check that out on your own, and skip straight to something a lot more interesting: a server that can send static files, not just a single line of text:

	var sys = require("sys"),
	    http = require("http"),
	    url = require("url"),
	    path = require("path"),
	    fs = require("fs");

	http.createServer(function(request, response) {
	    var uri = url.parse(request.url).pathname;
	    var filename = path.join(process.cwd(), uri);
	    path.exists(filename, function(exists) {
	        if(!exists) {
	            response.writeHead(404, {"Content-Type": "text/plain"});
	            response.end("404 Not Foundn");
	            return;
	        }

	        fs.readFile(filename, "binary", function(err, file) {
	            if(err) {
	                response.writeHead(500, {"Content-Type": "text/plain"});
	                response.end(err + "n");
	                return;
	            }

	            response.writeHead(200);
	            response.end(file, "binary");
	        });
	    });
	}).listen(8080);

	console.log("Server running at http://localhost:8080/");

Thanks much to Mike Amundsen for the pointer to similar code. This particular example was posted by Devon Govett on the Nettuts+ training blog, although it’s been updated for the current version of Node in a number of places. Devon’s entire tutorial post is actually a great companion piece on getting up to speed on Node once you have a handle on the basics.

If you’re new to Node, type this code into a text file and save the file as NodeFileServer.js. Then head out to the Node website and download Node or check it out from the git repository. You’ll need to build the code from source; if you’re new to Unix, make, and configure, then check out the online build instructions for help.

Node runs JavaScript, but isn’t JavaScript

Don’t worry that you’ve put aside NodeFileServer.js for a moment; you’ll come back to it and more JavaScript shortly. For now, soak in the realization that you’ve just run through the classic Unix configuration and build process:

./configure
make
make install

That should come with another realization: Node itself isn’t JavaScript. Node is a program for running JavaScript, but isn’t JavaScript itself. In fact, Node is a C program. Do a directory listing on the Node/src directory and you’ll see something like this:

Click to enlarge

For all of you thinking that JavaScript is a poor language in which to be writing server-side tools, you’re half right. Yes, JavaScript is not equipped to deal with operating system-level sockets and network connectivity. But Node isn’t written in JavaScript; it’s written in C, a language perfectly capable of doing the grunt work and heavy lifting required for networking. JavaScript is perfectly capable of sending instructions to a C program that can be carried out in the dungeons of your OS. In fact, JavaScript is far more accessible than C to most programmers — something worth noting now, and that will come up again and again in the reasons for looking seriously at Node.

The primary usage of Node further reflects that while Node works with JavaScript, it isn’t itself JavaScript. You run it from the command line:

 — (bdm0509@Bretts-MacBook-Pro Sun, 29 May 11) —  —  —  —  —  —  —  —  —  — (/Users/bdm0509/tmp/Node/src) —
 — (09:09 $)-> export PATH=$HOME/local/Node/bin:$PATH

 — (bdm0509@Bretts-MacBook-Pro Sun, 29 May 11) —  —  —  —  —  —  —  —  —  — (/Users/bdm0509/tmp/Node/src) —
 — (09:09 $)-> cd ~/examples

 — (bdm0509@Bretts-MacBook-Pro Sun, 29 May 11) —  —  —  —  —  —  —  —  —  —  —  — (/Users/bdm0509/examples) —
 — (09:09 $)-> Node NodeFileServer.js
Server running at http://127.0.0.1:1337/

And there you have it. While there’s a lot more to be said about that status line — and what’s really going on at port 1337 — the big news here is that Node is a program that you feed JavaScript. What Node then does with that JavaScript isn’t worth much ink; to some degree, just accept that what it does, it does. This frees you up to write JavaScript, not worry about learning C. Heck, a big appeal to Node is that you can actually write a server without worrying about C. That’s the point.

Interacting with a “Node server”

Make sure you still have your NodeFileServer.js code running via Node. Then you can hit your local machine — on port 1337 — and see this unremarkable output.

Click to enlarge

Yes, this is about as mundane as you can get. Well, that is, until you realize that you’ve actually written a file server in about 20 lines of code. The output you see — the actual code of the script you wrote — isn’t canned in the script itself. It’s being served from the file system. Throw an image into the same directory, and simply add the name of the image to the end of your URL, like http://localhost:8080/my_image.png:

Node happily serves this binary image up. That’s pretty remarkable, when you again refer to the brevity of the code. On top of that, how hard would it be if you wanted to write your own server code in JavaScript? Not only that, but suppose you wanted to write that code to handle multiple requests? (That’s a hint; open up four, five, or 10 browsers and hit the server.) The beauty of Node is that you can write entirely simple and mundane JavaScript to get these results.

A quick line-by-line primer

There’s a lot more to talk about around Node than in the actual code that runs a server. Still, it’s worth taking a blisteringly fast cruise through NodeFileServer.js before moving on. Take another look at the code:

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello Worldn');
}).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');

First, you have a call to a function called require(). The use of require() has been a long-standing request by programmers. You can actually find this mentioned in some of the discussions on JavaScript modularity, as well as germane to CommonJS, and a pretty cool implementation by O’Reilly author David Flanagan from 2009. In other words, require() may be new to you, but it isn’t an untested, careless piece of Node code. It’s core to using modular JavaScript, and something of which Node takes heavy advantage.

Then, the resulting http variable is used to create a server. That server is handed a function block to run when it’s contacted. This particular function ignores the request completely and just writes out a response, in text/plain, saying simply “Hello Worldn”. Pretty straightforward stuff.

In fact, this lays out the standard pattern for Node usage:

1. Define the type of interaction and get a variable for working with that interaction (via require()).
2. Create a new server (via createServer()).
3. Hand that server a function for handling requests.
   • The request handling function should include a request …
   • … and a response.
4. Tell the server to start handling requests on a specific port and IP (via listen).

Lost in translation

Despite the ease with which you can get a server coded in JavaScript (regardless of whether the actual code-running facility is C or anything else) still begs the question: Should you write a server in JavaScript? To really get a handle on the answer to this question, consider a pretty typical use case.

The JSON round trip

You’ve got a typical web application, HTML front-end with CSS styling, and JavaScript for the validation and communication with a server. And because you’re up on the interactive web, you’re using Ajax and not relying purely on a form’s POST to get data to and from the server. If this is you, then you’re probably comfortable with JSON, too, as that’s the almost de facto means of sending data across the wire these days.

So you’ve got an Ajax request that says, for example, “give me more information about some particular guitar on an online auction site.” That request gets thrown across the network to a PHP program running on a server somewhere. The PHP server has to send a lot of information back to the JavaScript requestor, and it’s got to send that information in some format that JavaScript can unpack. So the return information is bundled up into an array, which can then be converted to JSON, sort of like this:

$itemGuitar = array(
  'id' => 'itemGuitar',
  'description' => 'Pete Townshend once played this guitar while his own axe ' .
                    was in the shop having bits of drumkit removed from it.',
  'price' => 5695.99,
  'urls' => array('http://www.thewho.com', 'http://en.wikipedia.com/wiki/Pete_Townshend')
);

$output = json_encode($itemGuitar);
print($output);

Back on the client, the JavaScript gets this chunk of information, which has changed slightly because of JSON and transmission. The client basically gets something like this:

{
  "id": "itemGuitar",
  "description": "Pete Townshend once played this guitar...",
  "price": 5695.99,
  "urls": ["http://www.thewho.com", "http://en.wikipedia.com/wiki/Pete_Townshend"]
}

This is pretty standard fare. Then, it’s easy to convert this text “thing” into an object in JavaScript. You just call eval(), like this:

var itemDetails = eval('(' + jsonDataString + ')');

The result is a nice JavaScript object with properties that match up to the JSON array-like structure. Of course, since the jsonDataString usually is returned from a server, you’re more likely to see code like this:

var itemDetails = eval('(' + request.responseText + ')');

This is the typical JSON round trip. But there are problems here … big, big problems.

Subtlety and nuance destroy solid code

First, there’s a major problem in that this sort of code relies heavily on a translator. In this case, the translator is the JSON interpreter and related code, and there are in fact two dependencies: a JSON interpreter for Java in the form of what eval() does with the response text, and the JSON interpreter for PHP. As of PHP 5.2.0, that interpreter is included with PHP, but it’s still essentially an external dependency, separate from the core of PHP.

Now, this isn’t a rant about translation itself. There’s nothing to suggest that there are problems in taking, say, an “l” and turning it into an “i”, or something that’s item 1 in an array and reporting it as being item 2 in an array. There’s a lot of testing that occurs before JSON tools are ever released to ensure that what gets reported is correct, and accurate round tripping from a client to a server and back again are possible. Lots and lots and lots of testing is involved …

And that is in fact a problem.

The dependency of JavaScript and PHP (and C and Lisp and Clojure and Eiffel and … well, see the figure below for all the JSON toolkits floating around for a ton of different languages) on a toolkit is a huge issue. In other words, the problem isn’t the translation but the translator. While programming languages evolve slowly, the uses to which these languages are applied is growing quickly. The result is that JSON is being put to use in areas of complexity that simply didn’t exists or went untouched even a few months ago. And with each new iteration — each new depth of recursion and combination of data types — it’s possible that an area is discovered that the translator doesn’t support.

A selection of JSON toolkits. Click to see the full list

That’s not in itself bad. In fact, it argues for the popularity of JSON that it’s constantly put to new use. But with the “new” comes the “does it support the new?” So JSON has to evolve from time to time, and that means testing, and retesting, and release on tons of platforms. You, the programmer, may have to rearrange your data; or wait on a release to support your needs; or hack at JSON yourself. Again, many of these are just the so-called costs of programming.

But imagine you could ditch the translation — and therefore the translator — altogether. Imagine you could write, not JSON round tripping, but JavaScript end to end.

That’s the promise of Node. All the text you’ve just read — about PHP including JSON in 5.2.0 but not before, about arrays becoming objects, about data being configured in new ways and requiring new things from JSON — it all goes away when you have JavaScript sending data and receiving and responding to that data.

eval() in JavaScript is (Potentially) the Devil

As if that’s not enough reason to look seriously at Node, there’s the pesky issue of running eval() on a string. It’s long been accepted that eval() is dangerous stuff. It runs code that you can only see as textual data; it’s the equivalent of that “Run Your SQL by typing it in here” unvalidated text box, open to SQL injection and malicious intent. It’s quite possible that every time eval() is passed in a string, a puppy somewhere in the Midwest shivers and a mom on the Eastern Seaboard stubs her toe and curses. It’s that precarious. There’s plenty to read about online, and it’s not worth going into in detail here. Just Google “eval JavaScript evil” or “eval JavaScript injection” to get a good taste of the issues.

Still, Node without any context doesn’t allow you to avoid eval(), so there are potentially still shivering puppies out there. However, Node used as it’s intended absolutely gets you around the typical eval() problems. Node is often called evented JavaScript or evented I/O, and that little word — “evented” — is hugely important. But to get a hold of what evented really means, and why it gets you out of the dangers of eval(), you’ve got to understand not just how JSON is typically round tripped in applications, but how the very structure of applications on the web are typically architected.

Today’s web is a big-event web

Typical web forms are “big-event” submitters. In other words, lots of data entry and selection happens — a user fills out text boxes, selects choices from combo boxes, selects items from a list, and so on — and then all of that information is submitted to a server. There’s a single “big event” from the programming perspective: the submission of all that form data, usually through a POST. That’s pretty much how the web operated, pre-Ajax.

Sending lots of data at one time

With Ajax, there is a little more of what’s called evented programming. There are more events that trigger interaction with the server. The classic case is the entry of a zip code, and then a resulting call to the server to get a city and state. With Ajax and the XmlHttpRequest, tons of data didn’t have to be gobbed up and thrown to the server all at once. However, that doesn’t change the reality that the web is still mostly a big-event place. Ajax is used far more often to achieve interesting visuals, do quick validations, and submit forms without leaving a page than it is to create truly evented web pages. So even though a form isn’t submitting a big gob of information with a POST, an Ajax request is doing the same thing.

Honestly, that’s only partly the fault of less-than-creative Ajax programmers. Every time you send off a request — no matter how small — there’s a lot of network traffic going on. A server has to respond to that request, usually with a new process in its own thread. So if you really move to an evented model, where you might have 10 or 15 individual micro-requests going from a single page to a server, you’re going to have 10 or 15 threads (maybe less, depending on how threads are pooled and how quickly they’re reclaimed on the server) firing up. Now multiply that by 1,000 or 10,000 or 1,000,000 copies of a given page floating around … and you could have chaos. Network slowdown. System crashes.

The result is that, in most cases, the Web needs to be, at a minimum, a medium-event place. The result of this concession is that server-side programs aren’t sending back tiny responses to very small and focused requests. They’re sending back multiple bits of data, and that requires JSON, and then you’re back to the eval() problem. The problem is eval(), sure, but the problem is also — from a certain perspective, at least — the nature of the web and threading and HTTP traffic between a web page and a server-side program responding to that request.

(Some of you more advanced JavaScript folks are screaming at this point, because you know better than to use eval(). Instead, you’re using something like JSON.parse() instead of eval(). And there are also some compelling arguments for careful usage of eval(). These are things worth screaming about. Still, just see how many questions there are surrounding eval() on sites like Stack Overflow and you’ll realize that most folks don’t use eval() correctly or safely. It’s a problem, because there are lots of intermediate programmers who just aren’t aware of the issues around eval().)

Sending a little data at all times

Node brings a different approach to the party: it seeks to move you and your web applications to an evented model, or if you like, a “small event” model. In other words, instead of sending a few requests with lots of data, you should be sending tons of requests, on lots of events, with tiny bits of data, or requests that need a response with only a tiny bit of data. In some cases, you have to almost recall your GUI programming. (All the Java Swing folks can finally use their pent-up GUI knowledge.) So a user enters their first and last name, and while they’re moving to the next box, a request is already requesting validation of just that name against existing names. The same is true for zip codes, and addresses, and phone numbers. There’s a constant stream of requesting and responding happening, tied to almost every conceivable event on a page.

So what’s the difference? Why is this possible with Node, and aren’t the same issues around threading existent here? Well, no, they’re not. Node’s own site explains their philosophy the best:

Node’s goal is to provide an easy way to build scalable network programs. In the “hello world” web server example … many client connections can be handled concurrently. Node tells the operating system (through epoll, kqueue, /dev/poll, or select) that it should be notified when a new connection is made, and then it goes to sleep. If someone new connects, then it executes the callback. Each connection is only a small heap allocation.

Node has no blocks, no threads competing for the same resource (Node is happy to just let things happen however they happen), nothing that has to start up upon request. Node just sits around waiting (quite literally; unused Node responders are sleeping). When a request comes in, it’s handled. This results in very fast code, without uber-programmers writing the server-side behavior.

Yes, chaos can ensue

It’s worth pointing out that this model does allow all the problems that any non-blocking system allows to come into play: one process (not thread) writing to a data store while another one grabs just-invalidated data; intrusions into what amounts to a transaction; and so on. But realize that the majority of event-based programming on a web form is read-only! How often are you actually modifying data in a micro-request? Very rarely. Instead, there’s a constant validation, data lookup, and querying going on. In these cases, it’s better to just fire away with the requests. The database itself may add some locking, but in general, good databases will do this much more efficiently than server-side code, anyway; and they’ll certainly handle things better than an operating system will spin up and down threads for a generic, “a web response came in” process.

Additionally, Node does have plans to allow for process forking, and the HTML5 Web Workers API is the engine that will probably make this feature go. Still, if you move to an evented model for your web application, you’ll probably run into an issue where you might want threading in less than one out of 100 situations. Still, the changes are best in how you think about your web applications, and how often you send and receive data from a server, rather than in how Node works.

In the right place at the right time

There’s another web pattern at work here, and it’s probably far more important than whether you use Node or not, and how evented your web applications are. It’s simply this: use different solutions for different problems. Even better, use the right solution for a particular problem, regardless of whether that’s the solution you’ve been using for all your other problems.

The inertia of familiarity

There’s a certain inertia in not just web design, but all of programming. That inertia can be stated axiomatically like this: the more you learn, use, and become good at a certain approach or technique or language, the more likely you are to use that approach/technique/language widely. It’s one of those principles that sounds good until you dig deeply. Yes, it’s good to learn a language or toolkit well, and to employ it widely. But this inertia often causes you to use a tool because you know it, rather than because it’s the right tool.

Look at Ajax, something already discussed. Initially, Ajax provided a solid approach to sending quick requests, without form submissions, to a server. Now it’s become a drop-in replacement for all form submissions. That’s taking a technology, learning it, applying it, and then eventually over-applying it. There’s still a solid place for form submissions — when a form needs to be submitted! As simple as it sounds, there are tends of thousands of web applications submitting forms with Ajax, just because the lead web developer is up on Ajax.

In the same vein, it’s possible to get excited about Node — probably because you buy into all the splendid and wise observations you’ve been reading — and then use it everywhere. Suddenly, you’re replacing all your PHP and Perl back-ends with Node. The result? A mess. In fact, you’ll be forced to have several web forms do just what Node isn’t meant for: submit big chunks of data to JavaScript on the server via Node, and force that JavaScript to either send back a chunk of JSON that’s got to be parsed or eval()ed, or send back a full-blown HTML page or an HTTP redirect.

But that’s simply not what Node is best at. It’s great at micro-requests; at evented I/O. Use Node for quick communication between a web page and a server. Use form submissions to send big chunks of data to the server. Use PHP and Perl to do heavy database lifting and generate dynamic HTML pages. Use Node to provide a means for server-side JavaScript to run and handle small requests. Throw in Rails and Spring and servlets and whatever else you need. But make your decisions based upon the problem you’re solving, rather than what you happen to know best at the time.

Node’s promise of simplicity

There’s one last note worth making. When you take this broad approach to programming, you’ll often find that you’re not having to go as deeply into each toolkit, API, and framework you use. By using your tools for what they’re best at, you don’t need to be able to staple with hammers or measure with drills. Using tools for their intended purpose typically means you use the core capabilities more. So while you’re creating generalists — programmers that know lots of things — you are also reducing the need for specialists — programmers that know one or two things really, really well. Of course, every pointy-haired boss also realizes that those specialists are really, really expensive and hard to find.

Learning Node might take a little effort, but it’s going to pay off. Why? Because you’re afforded solutions to your web application problems that require only JavaScript to solve. That means your existing JavaScript expertise comes into play. And when you do need to use PHP or Perl — because it’s the right solution for a particular problem — you don’t need a PHP or Perl guru. You need to know the basics, and those needs can be expanded when the problem requires expansion. Stretching comes at the behest of new problems, rather than stretching poor solutions thinly.

Your biggest challenge is the continual move to a web that is made up of smaller pieces, talking more often, and the combination of what can seem like a dizzying array of technologies. However, taking the core features of 100 technologies is always going to serve you better than taking 100% of one technology and trying to solve 100 problems. Node and evented I/O isn’t a solution to every problem, but it sure is a solution to some important problems.

OSCON 2011, coming up later this month, will feature an entire day dedicated to Node. Learn more about Node Day and save 20% on registration with the codeOS11RAD.

Related:

• Node: Up and Running (Book)
• The secrets of Node’s success
• Why a JavaScript hater thinks everyone needs to learn JavaScript in the next year
• JavaScript spread to the edges and became permanent in the process
• What is Node.js and what does it do? (Video)
• What is HTML5?

Bear with me; anecdotes are required.

1990:

Twenty years ago, I was 13, and my father was not. He owned a 286, or perhaps a 386; I very much did not. For him, his computer was a functional employee. It did what he told it to do, slightly faster than a mathematical child prodigy, and he cared very little for what that manilla box chose to do when he was not around.

I, on the other hand, was far more interested in how that slab behaved, and the psychosis of that behavior. I wanted inside. And while I was forbidden to play with the computer, I was not forbidden to open an unlocked toolbox, find a Phillips-head screwdriver, and put it to work.

At some point, my unscrewing of the eyeless computer definitely became play for me. I opened the closed system, and have been working in, with, and around computers every since. I was yelled at soundly that evening — leaving the various parts scattered around the room and then going to a church youth function without reassembling those parts might have contributed to the problem — but it changed me.

Rules? Sure. I learn best when there are rules, because they beg me to break them and see what happens.

2000:

I was a Java and XML expert, and Jason Hunter was not. He hated SAX and DOM; I had been teaching them so long that I’d forgotten I hated them, too.

He didn’t like the rules. He thought the APIs were stupid. He was right. He convinced me to open the closed system, and JDOM was born. Nobody liked us in those early days; certainly not the W3C or Sun, who was busily endorsing SAX and DOM in their Java API stack.

Now, JDOM is a core part of a whole lot of Java and XML processing. There are quotes from Sun on the JDOM quotes page.

2010:

A lot of people are upset about how closed the iPhone, and now the iPad, are. Cory Doctorow — who I usually enjoy — wrote a lengthy piece about the evils of the iPad and its awful closed system.

He says, among other things, this:

The model of interaction with the iPad is to be a “consumer,” what William Gibson memorably described as “something the size of a baby hippo, the color of a week-old boiled potato, that lives by itself, in the dark, in a double-wide on the outskirts of Topeka. It’s covered with eyes and it sweats constantly. The sweat runs into those eyes and makes them sting. It has no mouth … no genitals, and can only express its mute extremes of murderous rage and infantile desire by changing the channels on a universal remote.”

The way you improve your iPad isn’t to figure out how it works and making it better. The way you improve the iPad is to buy iApps. Buying an iPad for your kids isn’t a means of jump-starting the realization that the world is yours to take apart and reassemble; it’s a way of telling your offspring that even changing the batteries is something you have to leave to the professionals.

First, I agree completely with Cory’s premise. I agree that Apple has taken far too much away. I agree that it is infantalizing to require us to send in the iPad to get its battery replaced. I agree that we should not have the App Store as a great gatekeeper in the sky cloud.

But, my gosh, when did developers ever need permission to break things? When did Steve Jobs become not just rule maker, but some sort of deity that actually prevented me from ignoring said rule maker, and doing whatever I could with my device?

Again, from Cory:

Then there’s the device itself: clearly there’s a lot of thoughtfulness and smarts that went into the design. But there’s also a palpable contempt for the owner. I believe — really believe — in the stirring words of the Maker Manifesto: if you can’t open it, you don’t own it. Screws not glue. The original Apple ][+ came with schematics for the circuit boards, and birthed a generation of hardware and software hackers who upended the world for the better. If you wanted your kid to grow up to be a confident, entrepreneurial, and firmly in the camp that believes that you should forever be rearranging the world to make it better, you bought her an Apple ][+. [Note: Link added.]

I’m sorry, but this is revisionist. While the Apple ][+ might have enabled a generation to follow, the hackers were well alive before they were handed schematics. Case in point, my earlier anecdote: I needed neither permission nor instructions from my father (or IBM) to break into that enticing bland metal case and tear out its guts. I just needed a screwdriver.

In fact, there is a vast difference between an intention for a consumer to not open a device and an inability of that device to be opened. Thankfully, iFixit didn’t get the memo that Steve would be upset if you broke into the iPad, and opened it within hours of release.

Then again, perhaps they did get the memo. They just chose to ignore it.

Mike Loukides, a fellow editor and one of the brightest minds at O’Reilly, said it well (in email):

… the iPad presents a challenge, and that’s a good thing. The argument is perverse (closed is good because it invites you to hack it), but I think it’s valid.

Yes, it’s perverse. And I, like Cory, am strongly for Apple getting out of my way. I’d like things to be more open. I’d like to have an easier means of sharing my comics, and my books, and my data. (I actually think that is the strongest portion of Cory’s argument, and one I firmly agree with.)

But, failing Apple’s permission, I’m sure there are many, many 13-year olds, unafraid of dad — or perhaps very afraid, but willing to pay the price — and they are picking up their screwdrivers.

Resist. Why not? It’s how creativity is born.

We’d all grimace at this abuse of the web (and even the most common sense of design). Why? Well, besides the tag itself being obnoxious, this was a classic case of taking your content and manually controlling how that content would look. That little bit of data — “40% off” — was inexorably and permanently linked with a bit of formatting — the <blink> tag.

And you all know about this, whether your knowledge is localized to the blink tag, or just produced in a growing separation-of-content model for web design and development. Hardly anyone intentionally and consistently mixes content and presentation in web pages these days. CSS (and SASS, in some circles) simply makes it too easy to keep your style separate from your content.

So here we are in 2010, all design-sophisticates, separating our content from our style. Well, on the web we are. The more I listen and watch and involve myself in ebooks, though, the more I find myself thinking about the blink tag again. And while I think the term “ebook” is useful and possibly necessary for intelligent conversation, I just wonder if that little “e” in front of “ebook” might be on its way toward becoming the new blink tag.

I won’t draw this out. I’ll make it simple. Right now, I’m typing into Movable Type’s editor. I’m typing words, and sentences, and paragraphs. And MT will take those words, sentences, and paragraphs–my content–and display it on the Radar blog, formatted, styled, easily consumable by web browsers and RSS readers.

At no point has it even occurred to me, until right now, that I’m in fact typing e-words or e-sentences. I’ve not thought about adding an e-carriage return to separate this e-paragraph from the next e-paragraph.

How absurd would that be? Come on. I’m just typing. I’m creating content. Plain old raw content. And it just so happens that my content will be delivered in a particular format (a blog). I suppose if O’Reilly wanted, they could assemble this content with a bunch of other content, and print the sum of all that content into a book. It wouldn’t be an e-book because its content started out digital. That’s just as silly as saying that because content first lived in printed form, and then was released digitally, we have a digital print book.

So why the “e” in ebook? Yes, I know there are some naming issues. We have to use language, and we need some means of distinguishing a book bound with glue, printed on paper, from a book that lives purely in the digital realm. I get that with every bit of my editor-laden, grammarian being. Communication requires a distinction.

But I’m increasingly seeing the ebook treated as more than just a language distinction. I see people creating content with a specific display paradigm in mind. The content assumes a certain width of screen; a certain font size. Images are being inserted not because they belong, but “because iPhone and iPad users will expect more imagery.”

Really?

So just because your image is easier to look upon than a blink tag, are we not returning to a very bad time in the history of the Internet?

So let’s be clear: I’m saying something that isn’t overly original, but bears saying (again). The first group/publisher/company/person who moves away from the ebook and to content–content that can be delivered to a variety of media, digital and non-digital, with display and style applied separate from and after content creation–wins. They’ll have lower costs involved with taking content and making it available on multiple platforms. They’ll have content that can adapt to new formats quickly, because there’s no “un-presenting” content before it can be repurposed for another platform.

Sure, we’ll always have ebooks. But can we all hope that this becomes a term based on a distinction in display format and medium, rather than a fundamental distinction between one type of content and another? I’m really not up for another blink tag.

I’ve recently resumed a childhood love affair with comics. In particular, I’m a fan of the Uncanny X-Men. While they’re not as edgy as the Dark Knight, and not as hip as a Dark Horse mini-series, they’re what got me started on comics, and what I continually go back to. (Besides that, they’re much more interesting and generally less sucky than the movies and cartoons of the same name.)

Of course, it’s been a while, so I hopped over to UncannyXMen.net to figure out what’s been going on. They have a nice primer to help you figure out how all the various titles intersect, which is non-trivial to keep track of in the X-Universe.

Interestingly, I ran across this:

This struck me: continuity. Readers loved the continuity of the story.

While it’s easy to chalk this up as a function of good fiction, I don’t think it’s that easy. Putting aside issues of story, I’m struck by how much looking back and forth I tend to do in reading a comic. I’m scanning a bit ahead, and reflecting back on what I just read and saw, even while reading the current panel. I’ve got this constant sense of context; I have a continuity in which what I’m learning (about a comic book character, about a love interest, about an island that’s about to be submerged by supersonic waves triggering earthquakes along fault lines, etc.) fits.

So why would we simply accept that in non-fiction–especially projects and products that purport to actually teach something–we can’t have continuity?

In many ways, this is the genius of visual series like Head First, and to a lesser degree in this specific case, the Missing Manuals. I’d also argue that this visual format does wonders for the Twitter Book and our new Best iPhone Apps book and site. Without having to re-read a page or flip ahead, you have a sense of visual context. You have a continuity that can be absorbed in a glance, even if you’re ready body text at the top of a right-hand page.

I could go on and on, but let’s stop the exposition. Here’s a simple question: in your reading, your writing, your speaking, your programming, what are you doing to create and absorb context and continuity? I believe there are ways to achieve this in almost every field, and I believe this is an important part of what sets the elite apart from the… well… non-elite, in terms of communication.

Where’s your continuity?

So what’s missing here? Well, it would seem obvious… except to many technical book authors. See, for most folks, the obvious answer here is, “There are no students!”

But for the average technical book author — and to be clear, I’m one of that crowd, so I’m speaking personally and from experience — we would all, loudly, cry out, “There’s no teacher!”

What a fundamental disconnect.

See, those of us who write do that writing alone (or, in some cases, relatively alone. That’s author-speak for, “at my local Starbucks with earphones”). And yet, we’ll quickly call ourselves “teachers.” But what other type of teacher functions without a group of people in front of them, or at least in mind?

Can you imagine a new math teacher walking into a classroom, and gaping at all the kids seated in the room? “What are all these kids doing in here?” We’d very politely usher that teacher back out into unemployment (or perhaps into further training).

Take a short step away from the empty classroom, and consider the dying technical book market. Yes, it’s dying. Sales are plummeting for traditional technical books. We’re having to examine other markets, other formats, other approaches…

And we’re back to the empty classroom. Right? Because we’re building screencasts with an anonymous teacher speaking into a microphone in an empty room. Or we’re writing iPhone apps that are beautiful… and do little more than highlight the teacher’s knowledge.

But what if… what if we looked at that picture above, and realized that what’s missing is the student. Better, the learner.

Socrates suggested that learning is remembering (I’m simplifying, I know, but I’ve already written longer than I should have). And the beautiful part of his picture of learning was suggesting that he was less a teacher than a midwife. He basically didn’t teach; he instead aided the learner in learning. He was a facilitator, little more.

So one of the things you’re going to see over these next few weeks — from me and from O’Reilly — is a renewed focus on the learner. We’re going to write and ask questions about how to facilitate, rather than lecture. We’re going to push out some new and engaging products and ideas (some for-pay, some for-free), and we’re going to put the focus on the learner.

I hate writing a declaratory piece like this, because it doesn’t encourage interaction as much as I’d prefer. In a sense, I’ve broken my own rules, and become lecturer instead of facilitator. I haven’t been a good midwife. I’m hoping you see that I’m trying to do a little stage-setting, for a lot of facilitation.

Maybe you’ve got comments, ideas, and thoughts. How do you do this? How do you keep your focus off your own “brilliance” and on your learner’s needs? What are you doing to keep the focus on where it belongs?

[A few important insertions: I'm not supposing that to respond to philosophical ideas about knowledge, education needs to change. I am suggesting, though, that a responsible response entails understanding the arguments, and either adhering to them, or forming a sound counter-argument to explain abandoning them.]

Two theses in particular caught my attention. First, Thomas Reid makes this astounding statement (cited from Thomas Reid’s Inquiry and Essays (1975), p. 275):

Another first principle is–That the natural faculties, by which we distinguish truth from error, are not fallacious.

What does this say? It says that our natural senses don’t tend to fail us. Now, I know, many of you are freaking out over this quote, especially in light of particle physics or molecular biology. And we can argue that over a good cup of coffee. But I will suggest that Reid is right in the macro-world.

I see a piano falling, I rightly assume several things:

1. It is indeed falling, and I am not instead rushing up toward it.

2. That piano is dangerous to be under, given that it’s falling.

There are plenty of other observations, but you get the idea.

So why does this matter? Well, how much do we allow the learner’s senses (and by senses, I don’t mean “ears listening to 90 minutes of lecture”) engage in a typical learning environment? How much do we allow the natural faculties to assert themselves, create knowledge, and then refine and provide context for the knowledge already gained?

I think it’s an important question. I think Dan Meyer is a master at this (check out his blog for some amazing examples of using pictures to stimulate learning). How are you doing this? What effect on education would an increased (as in, significantly more than you’re currently doing) amount of sensory learning create?

Let me know what you think. Oh, and as for my other philosophical quote that I think is important? More on that later this week…

There is good evidence that we “erase” and “rewrite” our memories every time we call them, suggesting that if it were ever possible to erase specific content, playing it back first might be an essential component.

This is a staggering statement. Consider the implications: when you recall a memory, you are capable of – and prone to – rewriting that memory in some form. I find this particularly fascinating in terms of teaching in a spiral method, something I continue to find effective and even critical in highly technical topics.

Take memory management in any programming language. It’s simply foolish to unload the truck on an unsuspecting learner, dumping out everything there is to know about memory management at one time, in one place, with little or no functional motivation. The better approach is to incrementally teach the topic, adding additional resolution, detail, and expansion only when new functionality is needed or additional understanding is required. In this way, you’re catering to the learner: each piece of information you’re unpacking is motivated by a need in that learner. This results in greater internalization of the information, and less information is categorized as “I don’t really need this. I’ll dump this.”

But with the quote by Aamodt and Wang, there’s another component at work here: earlier memories are potentially being rewritten as new learning takes place. This is intuitive, even: consider how often we mix up events that are very similar, but not the same. Have you eaten at Chuy’s 10 times in the last month (I’m about there)? If so, I’d suspect you’ll have a hard time distinguishing at which instance in 10 a certain conversation happened, especially without other mitigating details (a really close friend attended only one meal, or something particularly disastrous happened at another). Is it possible that the brain is trying to shove these similar events into one giant event, because we’re recalling an earlier (similar) event, replaying it, and rewriting it with the new one?

What this seems to suggest — and I grant that there’s a lot of theorizing and speculation happening, but what else is Radar good for if not some provocative thought — is that we must be extremely careful with context. When you recall an earlier mental model of something, and then augment that model, you may be rewriting the earlier model. In other words, you’re not just adding to an in-place model, but in fact replacing an earlier model with a newer, expanded one. So what are you doing to ensure the foundational models stay intact? Are you repeating the earlier model, and adding resolution? Or are you just writing about the “new stuff” without regard for the existing material?

I think most textbooks and technical books continue to heap on, assuming that pre-existing models remain in place. Foundational concepts never die, these books would assert (if not implicitly, then by the manner in which they teach). But perhaps those concepts do die! Perhaps this is why you may be adept at releasing memory or allocating memory, but would flail about helplessly at explaining what’s really going on. Is it possible that your original mental model has been overwritten, or even functionally replaced?

It’s an interesting thought. Context becomes critical, not just as a reminder of pre-existing material, but actually to ensure that pre-existing material is not lost altogether.

C’mon teachers, you must have thoughts on this… let’s hear them.

Yes, there he is… our glorious thought-leader, riding a jet ski. But Tim needs your help… seriously. Here’s the problem:

A few weeks ago, I was chatting with Tim. He mentioned that he’d recently taken his first ride on a Jet Ski. Several torturous minutes later, he got off, still alive and capable of detecting faint signals. But his back was suffering… badly.

Tim, as Tim is prone to do, let the ski rental place know of his pains. The instructor/rental guy looked at Tim, and simply said, “Oh. You need to lean forward.”

At this point in our conversation, Tim rolled his eyes, gave a half-wave of his hand, and said, “Oh, thanks. That would have been nice to know before I got on the ski.” Obviously, if the instructor had told Tim to lean forward before he took on the Old Man of the Sea, Tim’s back would have been saved a lot of hardship.

Or would it?

Along with all the other instruction Tim would have received, he’d have been told, “Oh, and be sure and lean forward.” Would this have stuck with Tim? Would it have been held up in his brain as important as, say, “Keep a tight grip on the handlebars?” Would it have competed with, “Look here… this is the ignition key. Turn it to start, turn it again to stop.”

Would simply telling Tim ahead of time to lean forward been enough to save Tim’s back?

Better yet, how would you have prevented Tim’s back pain? Here’s the question, broken down for easy answering…

1. WHEN would you have told Tim, “Lean forward?”
2. HOW would you have told Tim?
3. Free response: what else would you have done/not done to ensure Tim got off the jet ski happy, healthy, and not hurting?

Come on… Tim’s back is counting on you figuring out how humans learn… how best to communicate… and what our brain does with information that is important, but maybe non-obvious in application or significance.

I’ve long heard people complain about how commercials represent the basest forms of humanity. Yesterday, I was reminded of this again, as Twitter was all ablaze with people in outrage over the latest GoDaddy.com commercial run during the Super Bowl.

All of this tends to make me roll my eyes a bit, and go, “Well, duh… of course commercials represent the basest forms of humanity. That’s why they work!”

Science has long told us that the brain is largely indiscriminate in terms of forming and retaining information (what we tend to simply call “memory.”) For instance, if you like and remember the song that played when you proposed to your wife, you’ll also tend to remember where you were when you heard the song, what the weather was if you were outdoors, details about what your intended was wearing, maybe even what you ate. That’s because along with the song itself, your brain carried along all the accompanying context.

The thing is, the brain’s not really that smart. It doesn’t realize that the two eggs on your plate in the diner really aren’t as important as the beautiful girl sitting across from you when you heard that new Norah Jones tune that made her go wild and accept your marriage proposal. In theory, those eggs don’t matter… nor does the awful plaid shirt you’d spilled coffee on. But the brain can’t discriminate. Instead, it just throws everything into “memory.”

Super Bowl ads prove this, once and for all. With ad revenue topping 206 million bucks (in a recession!), we were hit with ad after ad, all having next to nothing to do with the actual products in question. That’s because, well, the ad guys are smart. They payed attention to all the available data, which essentially says it’s better to trick the brain than to appeal to it rationally.

What works better? Slapping a logo across a stretched t-shirt, curving around some surgically enhanced porn star’s chest, or lauding the wonderful features of hosting at GoDaddy.com? Getting everyone to laugh at the fire-breathing/belching dude on a date, or trying to convince the world that the Budweiser hops are better than the Michelob hops?

It’s a trick. And, once again… duh!

So here’s your “the brain is stupid” exercise for today. Call it a little reverse engineering.

1. What Super Bowl commercial from this year do you best remember?*
2. What product was being advertised?
3. What Super Bowl commercial from last year do you best remember?
4. What product was being advertised?
5. What Super Bowl commercial do you best remember from any year?
6. What product was being advertised?

* Note the question: “best remember” as opposed to “like the best.” This becomes self-evident as you think back a few years. We remember things we don’t like in many cases.

I’ll bet good money that you’ll easily recall the answers to (1), (3), and (5), and that the products attached to each answer come pretty quickly after. Only in the poorest ad placement situations do the product lines not get carried along by our brains in the “making memory” process, due to our brain being indiscriminate about what is carried along by a memory.

Last question, and this is the important one:

Why did these commercial stick out? What made them “memorable?”

More good money says the commercials you remember have no correlation to products that are important to you, except by dumb luck. You’re reacting to the commercial, which in most cases has almost no relation at all to the product.

Let me re-ask the question like this: taking the three commercials you came up with above, what’s common? What do you think grabbed your brain by the throat (yes, the metaphor is breaking down. Just roll with me, here) and said, “Remember me!”?

How would you make a good Super Bowl commercial?