The iPad is the iPrius: Your Computer Consumerized

Eugene Shimalsky in his short piece "One Small iPad for Man, One Giant Leap for Apple" declares that the iPad is interesting primarily because it isn't a computer. As he puts it:

Yesterday, Apple got all of the geeks glued to their screens waiting for the "Jesus Tablet," iPad. An hour later, they were twittering that it did not come. Or maybe it just wasn't their Jesus?

It turns out it was his Mom's.

It's been a long time since most of us have used our computers to do anything approaching "computing," but the iPad explicitly leaves the baggage behind, leaps the conceptual gulf, and becomes something else entirely. Something consumery, media'ish, and not in the least bit intimidating.

The automobile went through a similar evolution. From eminently hackable to hood essentially sealed shut. When the automobile was new, you HAD to be a mechanic to own one. Later, being a mechanic gave you the option of tinkering and adapting it to your specific interests. In fact, that's how most people up until about 1985 learned to be mechanics. The big changes came with the catalytic converter and electronic ignition (and warranty language to match). Now the automobile has reached the point in its development where you don't even have to know whether it has a motor or an engine to use it, but to tinker at all requires highly specialized skills.

So, in some ways this evolution of the computer to the iPrius seems completely natural. I don't care all that much if the iPad is hermetically sealed, but I wonder uncomfortably if in a few years the MacBook and the PC will be too. Or, more likely, we'll just wake up one day to a world without MacBooks or PC's. As we continue our shift en mass to the mobile device ecosystem and the laptop as we know it goes the way of the desktop, banished to special purpose niches.

In mobile land, closed carrier heritage combined with Apple's product vectors may leave us with only closed options. A confluence of interests - commercial (get your pure non-pirated content only from me!), governmental (cyber defense!), and user (I want to be safe!) - will find that outcome attractive. Our generative and hacker-friendly world will be replaced by a sterile world of sealed aluminum.

No doubt the iPad will be hacked by someone to prove it is still possible. They'll run linux on it within a week of launch, but that's not where they will have learned those skills. They learned them on the highly generative PC they probably bought for something else. Slight differences in approachability and "ease of mastery" (as Zittrain puts it) make a big difference. The curves are steep. And tomorrow the people that buy iPad's descendants will be less likely to develop those skills. Who's going to buy a developer's license just to screw around?

For your phone Apple could make a strong argument that this kind of control was necessary. They needed to make sure it was reliable first and foremost as a phone (rather than reliable as a snooping device or wouldn't just crash every time you really needed to make a call). The argument is being extended to the iPad more because of Apple's culture than real need, and if I was Steve Jobs looking at iTunes receipts I would do the same thing. But... directionally this is a vector toward compuserve, not away from it. The iPad is Steve's Minitel terminal.

Just for the heck of it, imagine for a minute that the MacBookPro was locked up like the iPad. The apps that run on the iPhone have been mostly trivial. One person for a few weeks is probably the average effort. Eugene Lin may be willing to build apps on spec and hope for the best after they are submitted, but will Adobe? Imagine when Adobe invests $X millions building Lightroom for a year only to have it rejected because Apple launches Aperture the same week.

Skinner Box? There's an App for That

If you are reading this post it means that after countless misfires, I finally kept my attention focused long enough to finish it. That may seem like no big deal, a mere trifling effort, but I'm basking in the moment. In fact, I'll probably tweet it.

It didn't start out to be about digital Skinner boxes. It was a Radar backchannel email about the infamous Web 2.0 Expo Twitterfall incident. I got all curmudgeonly and ranted about continuous partial attention, Twitter as a snark amplifier, and the "Ignite'ification" of conferences (with apologies to Brady). In short, I demonstrated myself unfit to contribute to a blog called Radar.

I swear I'm not a Luddite. I'm not moving to Florida to bitch about the government full time and I'm not in some remote shack banging this out on an ancient Underwood. However, I guess I count myself among the skeptics when it comes to the unmitigated goodness of progress. Or at least its distant cousin, trendiness.

Anyway, I sent the email, inexplicably Jesse said "post!", and I tried reworking it. I still am. This piece has been grinding away like sand in my cerebral gears since, and along the way it has become about something else.

In The Anthologist, Nicholson Baker describes writing poetry as the process of starting with a story and building a poem around it. I try to do that with photography and build pictures around narrative and metaphor. After the work takes shape the story is carved back out and what remains hints at the story's existence, like a smoke ring without the mouth.

He says it better: "If you listen to them, the stories and fragments of your stories you hear can sometimes slide right into your poem and twirl around in it. Then later you cut out the story and the poem has a mysterious feeling of charged emptiness, like the dog after the operation." Don't worry about the dog, it lived and it isn't relevant. My point is that this post isn't about the Twitterfall fail story, that was just a catalyst. The inchoate uneasiness still twirling around in here is what's left of it.

This all began with these lingering questions: "Why are we conference attendees paying good money, traveling long distances, and sitting for hours in chairs narrower than our shoulders only to stare at our laptops? Why do we go to all that trouble and then spend the time Twittering and wall posting on the overwhelmed conference wifi? Or, more specifically, why are we so fascinated with our own 140 character banalities pouring down the stage curtain that we ignore, or worse, mob up on, the speakers that drew us there in the first place?"

As I kept working away on what has become this overlong post, the question eventually turned into, "why the hell can't I finish this?" This has become the post about distraction that I've been too distracted to complete. It's also about ADHD and the digital skinner box that makes it worse, narcissism's mirror, network collectivism and the opt-in borg, and an entropic counter-argument for plugging in anyway. So, here goes...

(continue reading)

Defense Department Releases Open Source Memo

I've been holding my breath for so long waiting for this memo that I may not remember how to start breathing again, but here it is. The Department of Defense Deputy CIO Dave Wennergren has signed and released "Clarifying Guidance on Open Source Software."

2009OSS

Written primarily by my friend Dan Risacher at the Office of Secretary of Defense the memo is intended to clear up common misconceptions and make it easier for DoD program managers to include OSS in their programs. Its goals are to improve agility, eliminate lock in, and reduce cost.

One of the memo's key points comes from Dave Wheeler at IDA - OSS is considered "Commercial Off the Shelf" software as far as DoD acquisition rules are concerned and therefore OSS must be considered on an equal footing by law whenever a program is doing market research prior to technology selection.

Some will argue that it doesn't go far enough by only encouraging and not demanding the use of OSS on government programs (I certainly have some sympathy for that point of view) but my hope is that this will at least provide some counter to the FUD machine - you know who you are - and keep moving OSS in defense toward a tipping point of acceptance.

By the way, if you are interested in open source in government and are in or near DC, make sure you check out GOSCON next Thursday, Nov 5. Dave Wennergren will be giving the breakfast keynote and you can ask him all about this memo.

Three Quick Open Source in Defense Links (and then one other)

Next week I'll be participating in the inaugural Military Open Source Software Working Group Conference in Atlanta Georgia. Open source conferences that focus on the defense market are often salesy, have a dearth of actual developers, and tend toward sartorial blandness - a sea of dark blue suits worn by open source vendor sales people so they can convince hesitant buyers that their wares are just like the other guys. Look, we even license it by the seat!

This grass roots event, which will be held at the Georgia Tech Research Institute Conference Center, is designed to answer the question raised by those other conferences; "where the geeks at?" It will even have a dress code to match, no suits allowed. There is still space available, so if you are having the kind of ridiculously cool summer that makes August in Atlanta sound appealing, pack your shorts and sandals and head down.

If you aren't familiar with the defense software space, it buys and builds an immense amount of software. Quite a lot of it is actually pretty cool too because it is designed to solve interesting problems. We're still waiting for the defense market to have its IBM/Apache moment, but when this market inevitably tips hard into open source the impact is going to be tremendous. Open source methods and licensing will be a conduit for technology transfer from the DoD into commercial use on a vast scale. However, what I think is really cool, is the opportunity it will offer for important participation in the other direction.

A couple of projects at the vanguard of this trend that just opened up are FalconView and Open CPI.

FalconView started life as a moving map for USAF mission planning and was already a great example of user innovation in the military. Recently the team at Georgia Tech took the next logical step and open sourced the bulk of the project.

My colleague John Scott (@johnmscott) and his team at Mercury Computer Systems just opened up the distinctly different Open CPI project. Sort of a middleware for FPGA's, it grew out of the signals processing field and, if it picks up community support, should make it simpler to develop and build hybridized hardware platforms for special purpose applications. I've written before at Radar about the trend in some areas away from pure commodity hardware in areas where performance and energy consumption are a priority. I think projects like Open CPI will contribute to this trend by making the development of specialized platforms more approachable.

This last link isn't related to open source software except for the fact that Gunnar Hellekson @ghelleks of Redhat pointed me to it. We were chatting over lunch about the epidemiology of virus and vulnerability propagation and the fact the removal term is too low to keep populations small. All too often, once a system on the network (whether in the enterprise or at the home) is infected, it stays infected until it is removed from the network and (hopefully responsibly recycled) sometime after it has been fully depreciated.

Furthermore, in a large enterprise with as many as millions of machines to deal with, it is simply impossible to manage the process of consistently hardening machines to prevent infection in the first place. If Population = (rate of infection - rate of removal)*t you can see that these two issues conspire to help the bot herders and other nefarious characters keep populations large.

To deal with the second problem (and perhaps someday enable a solution to the first) NIST has been developing the Security Content Automation Protocol (SCAP). Basically it is an extensible XML schema for defining the hundreds of security configuration parameters and their values that need to be managed. Once defined and rolled into profiles, agents running on various platforms can implement the profiles automagically. In DoD parlance, this means that Security Technical Implementation Guides (STIGs) can be implemented broadly, efficiently, and, perhaps most importantly, in an ongoing manner.

The Hacker Ethic - Harming Developers?

On Monday Neil McAllister posed the question "is the hacker ethic harming American developers?" Slashdot picked it up and Tim forwarded it to the Radar list. As you might expect, it resulted in some spirited discussion.

James Turner kicked things off with this response (it has been slightly edited from its email form). After James lays out his argument I'll reply with my thoughts. Then we hope to hear from you. Let us know what you think.

I've worked in a lot of organizations that thought that the kind of rigid deforesting paradigms that Nayar is referring to were the magic bullet to keeping all three of the variable (dollars, time, features) under control. Without exception, all they did was get in the way and reduce the productivity of the most senior people to the level of the most junior. All of them exhibited some degree of failure, some catastrophic.

The India shops *love* methodologies like UML and the like, specifically because the problems have been reduced to a simplistic enough granularity that they can be doled out to junior-level staff, who may have only been onboard a few weeks because of the massive churn over there.

At least 3 times at 3 different companies, I've seen major pieces of work brought back in-house because the Bangalore team had fallen so far behind or proved so unable to get beyond the literal description of work that they were endangering the project. When you combine the time difference with a tendency to halt dead in their tracks as soon as they hit a stumbling block, it can be a recipe for disaster.

There are certainly some good Indian shops, and I know some outstanding Indian developers (most of whom have come to the States.) But I find Nayar's comments hilarious. It's akin to someone saying that American football players aren't employable in Jamaica because they aren't able to limbo well. Look at the most successful Web 2.0 companies today, most of them started as garage enterprises with a few talented developers, not a 60 person team of UML jockies following some Arthur Anderson project management program. Heck, look at Google Labs.

In huge projects, you obviously need some master planning and coordination to make sure the tracks meet at the right place to drive in the spike, but I don't see any effort being made these days to right-size the amount of project overhead to the needs of the projects. Instead we get a one-size-fits-all approach that smothers anything but the largest project in paperwork. Even some of the original authors of the Agile manifesto, when I've talked to them, point out that part of being Agile is picking and choosing the right components of project management that make sense for a given task.

Nayar's remarks are incredibly self-serving. "We're the best, because we can mindlessly follow some arbitrary and flawed development process." Or is he claiming that Indian projects do better QA? Not in my experience...

This entire debacle is representative of a problem I think is endemic in the industry these days, the inability or unwillingness to engage in rapid prototyping. Every successful project I've ever worked on (and I've worked on some fairly large enterprise-sized projects), we started by designing and coding a quick "throw-away" skeleton of the application, that let us look at how the thing worked, where the unseen warts were, and where the vendors had lied about their APIs, etc. This is the crucial and neglected stage in project design, one that most modern design paradigms ignore or actively discourage. Even Agile tries to jump in and start coding the finished application from the get go, although if project teams were willing to aggressively refactor (a tenant of Agile), early project work could be a rapid prototype (although the story model of scrum really doesn't fit well with this, unless you make the prototype a story...)

This is also something I've never seen an offshored team do particularly well...

Well, I'll be damned if I'm going to jump in and take the side that says hacking is bad for American programmers. First off, I don't need that kind of flame bait and second, I don't believe it. I think approachable programming is hugely important because that's how many people get into the field in the first place. However, my reaction to the article was very different than James' and I might as well try to explain.

I'm not going to take an opposing position, but it's not really an orthogonal position either. Maybe it has a power factor of about .7 or so. Here's my (also edited) response...

When I when I read McAllister's piece, at least some of it resonated with me. Before we were bought by a large firm, we were a small company that grew from nothing to 250 people, about 200 or more of them were programmers. So, a whole lot of my time over three years was spent hiring programmers and building cohesive teams that could deliver to our customers.

In our hiring we aggressively hired hackers into the mix. We wanted outside-our-industry thinking and we thought they brought in creativity. We called it "hiring weird, but not weird weird." Occasionally we pushed it to weird and a half. For our efforts we got creative problem solving and interesting (but frequently weird) dinnertime conversation when we travelled.

However, our pollyanna idea of "disciplined teams catalyzed with a bit of weird" didn't always work out.

That leads to the bit that resonated with me: the sense that hacker = distilled essence of American individualism combined with lots of ISTJ Myer's Brigg's Type Indicator. Individualism is a trait that I hold dearly, but it can make a cohesive team effort difficult if people are unwilling to suborn themselves to the goals of the team. Remember those tee shirts the football team always wore at your university? "There is no I in team?" I sometimes joked that I was going to make a batch that said "I'm the Me in team."

Maybe we were just growing fast and it was going to take more storming and norming than I had patience for, but at times it was a struggle to get everyone to see past their individual biases and focus on what we were trying to achieve, and we couldn't do what we were trying to do with teams of one.

But it really wasn't a hacker problem if hacker means self taught like McAllister implies. We had a lot of people with CS degrees and we used to talk a lot about whether and how their degrees had prepared them for their jobs.

Separate from the individualist approach to development, few of our recent graduates came to us prepared with the terminology and practices of any development approach (or engineering approaches like continuous integration etc.). They knew how to code, but not how teams coded.

At one point I gave a talk on agile software development to about 100 CS students at a university in Philadelphia and I asked them to raise their hands if they had ever done a team project with greater than two people on the team. I don't recall anyone raising a hand. Then I asked if they had ever covered development methodologies in their classes and a few acknowledged they had, but it had been abstract classroom stuff only. That part surprised me.

I'm not sure that the "sanctity of engineering" argument really makes that much difference. I have little faith in McAllister's scheme to do computer engineering instead of computer science coursework.

My undergraduate degree was in Mechanical Engineering and I can only imagine how useless I would have been to a firm that actually did engineering, and for mostly the same reasons. I knew how to take integrals and I still know the packing ratio of a hexagonal close packed material but I didn't know squat about how a complex machine actually got designed in a team setting. It's interesting to note that the Engineer in Training exam I took (a precursor to the professional engineer's exam) didn't probe my knowledge of team practices at all.

Maybe there just isn't time in an undergraduate degree to teach everything that an engineer needs to know. Plus, can you imagine the drop out rate in CS/CE if ITIL was a required course?

Since James mentioned Google I'll switch gears and muse about ecosystems for a moment... I guess I tend to bristle when I hear that everyone should just develop software the way Google does.

Google is to computing what LA was to Aerospace and Electronics in the early 60's. It's gravitational force attracts five sigma talent (probably a bunch of six too) in ways that the rest of us can only envy. More generally, Silicon Valley has had programmer talent flowing into it for the last twenty years the way Hollywood sucks pretty people out of the midwest.

Maybe it's not as obvious because you can't spot brains the way you can spot an oddly beautiful wait staff, but the valley has been the vortex of a talent-laden embarrassment of riches for a long time and, if you work there, you might not even notice it. However, I think that at some level this effects what kinds of processes work when you build software. I think it's at least a little part of the reason why an ERP system in a manufacturing town gets implemented differently than MapReduce (there are other reasons too having to do with software as product vs software as supporting infrastructure). Combine that with the very clear shared vision of "lets do something great and get rich together" thing that valley firms often have, and well, it's easy to see how smart people coalesce to build amazing stuff.

It's easy to denigrate Arthur Anderson's progeny or the offshore firms they compete with, but they do different work, with a different talent pool, for different ends, and with a very different set of personal and organizational incentives. Or, put another way, Kelly Johnson didn't build the SR-71 with General Motor's engineers, and General Motors didn't design the Chevy Cavalier with the Skunkworks' processes. However, even at the Skunkworks, Johnson's brilliant engineers did conform to a process and work together as a team toward a shared vision. And, conversely, I bet a lot of talent is left on the table at General Motors because of processes too restrictive in their attempt to remove all uncertainty.

So,... maybe it's possible that Google's (or the valley's in general) processes are appropriate to an ecosystem that, because of the intellectual environment and potential for riches, is rich in IQ and initiative. So it ends up feeling more "special forces" and less like "infantry regiment." And over there closer to the hump in the normal distribution curve, or in a different cultural environment outside of the valley, a different flavor of processes may be effective?

The counter argument to that, which I'll go ahead and provide, is that I once helped teach a team of engineers at midwestern defense contractor how to do agile development. The effect was amazing and immediate and their productivity and satisfaction went up tremendously; until their management freaked out and shut it down when they "perceived" that it created too much uncertainty in their processes.

Well, it's obvious that I don't know the answers here, so, with that, I'll stop thinking out loud.

What do you think?

The Web^2: it's Exponential, but is it Contracting or Expanding?

The theme for the Web 2.0 Summit this year is Web Squared. It is rooted in the idea that as the web morphs into less of a hub and spoke distribution model and more of a network of connected people and things, innovation and opportunity on it are growing exponentially. There has been a little bit of discussion on the Radar back channel about exactly what this means, or should mean, and Nat started things off with a thoughtful response that probably should be blogged as well. In particular he introduced feedback loops into the discussion, and with Nat's prodding, I decided to share my response to his email here. I've edited it a bit to make it a *bit* more cohesive, and while it isn't as structured as I would like, these are my thoughts about the exponential future of the web and a little bit about how that future might also impinge on the future of government...

I agree with Nat that feedback loops are a great mental filter to view the world. I read a little bit of Wiener and now I see feedback loops everywhere. Furthermore, what I like about them as a mental model, is that they help me understand the web at the ecosystem level rather than at the level of a specific technology. Wiener defined a cybernetic system the way engineers define a thermodynamic system. In thermodynamics, a system is closed if no energy crosses its boundary. A cybernetic system is closed when no messages or information cross. Since messages are the lifeblood of feedback these boundaries are important. As an example, open government stuff is so exciting to me because once computing systems connect between the web and government, the boundaries of previously isolated cybernetic systems (e.g. the people and its government) begin to be permeable. And once they are permeable to computing messages they will also be permeable to cultural signals that can create cultural feedback loops. That will cause state to change on both sides of the boundary. Two small isolated cybernetic social systems become one larger integrated one with new feedback loops in place.

Regarding the exponential theme, I'm not sure that innovation is progressing as an exponential over time - although, in fairness, I'm still working on my unabashed optimism credentials. But... In the 1920's automobile companies were springing up like crazy in America. It was the era before production methods became the dominant competitive weapon and anyone with a good idea for a better combustion chamber design or a valve train or a styling cue could still try their hand at building a car company. With access to tools, labor, and know how Detroit in the 20's was a very generative environment for automobile innovation. But by 1980 even DeLorean with a trunk full of coke couldn't afford the startup costs - a combination of more sophisticated design requirements and the changes in production scale economics made it impossible.

Are Data Centers the Economic Equivalent of Manufacturing Plants?

The interesting parallel with the web (or computing and software more generally) is that the rise of the data center as a key piece of competitive know how and, perhaps more importantly, capital cost. The question in my mind is whether utility computing enhances generativity, or by making it contingent on powerful interests, effectively stifles generativity in the long term despite the generative potential of the technology (I'm shamelessly borrowing the idea of contingent generativity from Jonathan Zittrain). And a related question, does the introduction of capital cost as a major factor in the eco system eventually make the web feel more like Detroit in 1980? Will it fundamentally change the web by tying it firmly to those who can access sufficient capital? (Google spent over $800m on data center capital improvements last year. That's a number that even makes the Defense Department wistfully declare "we just can't afford to do what Google does.").

Or, ...the electric utilities made innovation with electrical devices more possible, but it doesn't necessarily follow that utility computing will always do the same. After all, electrical utilities ship their power to us where we use it in situ for whatever purpose we want, but utility computing requires us to send our "loads" to them where it is much easier to implement perfect mechanisms of surveillance and enforcement. Homeowners associations used association charters to turn neighborhoods into little fascist fiefs and data centers have the potential to do the same with EULA's.

Scale and Concentration (or, is the Universe Expanding or Contracting?)

As scale on the web increases there are competing concentrating and generative factors at work (any of which might be exponential). The concentrating factors (need for capital, sophisticated expertise, ...) tend, like gravity, to collapse the system down on itself in a variety of ways. I don't mean that it becomes less relevant or makes less money, I mean that it ends up feeling more like AT&T in the 60's with centralized control and vested interests and strict contingencies on generativity. Just like Apple's oversight of the app store. On the other hand, the factors that tend toward expansion are feedback loops that span organizational boundaries, ready access to seed funding, standards for cloud computing that encourage true commodity availability of non-contingent generative environments etc.

Figuring out which force will dominate is like trying to figure out whether the universe will expand forever or eventually contract. The balance between the factors is quite subtle, depends on minute variations in initial conditions, and is very difficult to predict. But, we can still ask ourselves, "how can we influence the broader cybernetic ecosystem of the web to encourage policy, practices, cultural values, etc. that will promote generative expansion rather than scale-driven contraction?"

Exponential Effects and Social Structures

Shifting gears for just a moment, complexity science is the other idea I tend to come back to as a frame for viewing the web that, while not directly related to the exponential theme, is at least peripheral. The web is fascinating in the way it has become the cybernetic substrate on which both technical and social patterns are emerging. Stripes form on a zebra because "black" and "white" chemical messengers from adjoining cells interact with each other differently over distance. Out of that simple mechanism complex patterns emerge. The web is transport for human messages that don't decay with geo-spatial distance. This geo-and-time-independent messaging is enabling human "striping" that is no longer geo-ethnic dependent.

Within a geography the existing striping can become more severe as the web enables self-selected and self-reinforcing pockets of auto-propaganda that combine with social graph clusters; clusters that only infrequently span value systems. The situation is reminiscent of 1930's era Spanish political parties and their newspapers, but operating at photo-multiplier tube speed. We consume the stuff that reinforces our world view and segregate ourselves into more and more thoroughly strident neighborhoods of belief. We remain physically in our geo-defined country, but in our chosen echo chamber we each live a very different intellectual and emotional experience in a whirlpool of exponentially hardening world view. Perhaps someday we'll live in "nation states" that are stripes of psychographic and value alignment instead of stripes in geography.

Of course, it's true that as long as we are physical beings we will continue to stripe locally in our physical world. The cybernetic overlay in human relationships provided by the the web doesn't replace that reality, but by augmenting it and letting us stripe along lines of affinity and value system without regard to geography, it contributes to fissures in our geo loyalties. These fissures are important because States exist to govern the physical world (trade, law, taxes, defense...) but they depend on shared values and culture to function effectively. Just look at Iran today to see the effect of incongruent value systems on co-located peoples.

Forge.mil Update and DISA Hacks Public Domain

On Monday DISA's forge.mil got another mention on Slashdot. Not really new news, but I think it has been getting press again because of the related news that DISA is also open sourcing its Corporate Management Information System (CMIS). CMIS is a suite of HR and related projects and DISA signed an agreement with OSSI to open source them.

I had been meaning to touch base with Rob Vietmeyer at DISA anyway and the Slashdot mention (plus a subtle kick from Sara Winge) got me off the dime. We are working on a project that we want to share across DoD and since Rob is the force behind forge.mil, I had been meaning to ask him about its uptake. I thought I'd share his answers here.

Since forge.mil was launched it has grown to about 1400 registered users and approximately 10% of them are active on any given day. There are approximately 70 active projects right now in a variety of categories. There are system utilities, geospatial systems, a control system for UAV's, an embedded engine control module, command and control system components, some SOA piece-parts for Net Enabled Command and Control (NECC) and Consolidated Afloat Networks and Enterprise Services (CANES), and sundry others. Project-related traffic (commits, downloads, etc.) is growing and there is a backlog of new projects being on-boarded (including at least one really high profile system that is looking to get broad participation).

What interested me about that list was that the code ranges across domains and from small niche items to components of large scale programs.

At this point most of the code seems to be licensed as "DoD Community Source" and a few projects are under Apache and BSD style licenses. DoD Community Source basically means that the code is "unlimited rights" or "government purpose license rights" under the Defense Federal Acquisition Rules (DFARS). While not "open source" in the OSI sense of the term, hosting code licensed this way on forge.mil should make collaboration across DoD the default rather than the exception. Basically these aren't copyright-based licenses but are designed to operate as though they are in practice - the goal is to do open source-like development within the DoD garden walls.

The source that is licensed under Apache / BSD style licenses is in fact licensed copyrighted material, but at this moment it is still difficult for non-DoD community members to participate because of forge.mil access limitations. DISA is looking into ways to mirror these open source materials to sourceforge instances outside of the DoD garden walls and to extend community participation across those boundaries as well. I think mirroring the code will be a lot easier than figuring out how to do boundary-spanning community.

Projects wanting to be hosted on forge.mil go through a "project adjudication" process that screens out the people just looking for a repository but who don't understand open (or, understand it but don't want it). Projects that don't want to provide open access to other DoD participants have been turned away.

I think there is something interesting hidden in plain view in that CMIS news as well.

One of the oddities of code written by government employees is that the government doesn't create copyright. In an ironic twist, this means that the government can't directly release code under open source licenses, since those licenses rely on copyright law to enforce their terms.

CMIS was written by government employees so DISA and OSSI had to figure out a hack to license it under a copyright-based license. Under the terms of their agreement DISA is releasing the code to OSSI under public domain, then OSSI is re-releasing a "derivative work" under OSL/ASL licenses.

I understand what DISA / OSSI is doing here but I wonder how much they've changed the code to make the "derivative" distinction. It's probably moot though because, assuming community forms around the stuff, it shouldn't take too long before a chain of real derivations is in place that would make the OSL/ASL license terms defensible.

Google's PowerMeter. It's Cool, but don't Bogart My Meter Data

Last week I read this piece in the New York Times about Google's PowerMeter, their entry into the smart meter game. The story was picked up in quite a few places but neither the NYT piece or related articles from other outlets expanded much on Google's underlying press release. Google's FAQ isn't very satisfying either; it has no depth so I didn't really know what to make of it. When I finished reading it I was left with an inchoate unsettled feeling and then I forgot about it. But on Friday evening I had a random conversation about it with a colleague who works in the meter data management (MDM) space. By the time we were through talking about what Google might be doing I had arrived at a position of love / hate. I'll explain the love first.

In terms of the attention this brings to energy consumption at the household level, I really love what Google is doing with this initiative. As they put it:

"But smart meters need to be coupled with a strategy to provide customers with easy access to near real-time data on their energy usage. We're working on a prototype product that would give people this information in an iGoogle gadget."

I agree completely. It's not exactly the same thing, but I've been amazed by how much my behavior behind the wheel changed once I started leaving average mpg permanently visible on my car's dashboard display. In short order I went from speed racer wannabe to one of those guys that gets harassed by co-workers for driving too slow. "Hey, can you hypermile on the way back from lunch? I'm starving."

While I am not sure that a gadget on the web will have the same right-there-in-front-of-my-eyes impact that my car's LCD display has, I'm convinced that Google has hit on something important. After all, today most of us have no idea how many kilowatts we use, what we use them for, or how much we're paying per kilowatt. We use power in our homes the way I used to drive my car.

Unfortunately, Google's FAQ doesn't really answer any questions about how the service works. But from statements like "Google is counting on others to build devices to feed data into PowerMeter technology" we can deduce that Google is proposing to correlate the total power reported by your smart meter with the data collected from individual loads inside the home. This is really cool, because not only does it make the information more generally accessible to you (in an easily accessible gadget), it proposes to tell you what it is in your house that is using that power, and when.

Google can do this because many national and state governments have begun to mandate smart meter programs. Most of us will probably have one on the side of our house pretty soon (especially if the stimulus bill speeds things up). Smart meters improve on their predecessors by automating meter reading, reporting consumption in intervals (typically 15 minutes), and they can send "last gasp" failure notifications in the event of power outages.

But, just like their dumb ancestors, they will be owned by the utility. This means that the data generated will ultimately be under control of the utility and hosted in their systems. The meter will talk to a utility data collector and from there its data will enter the utility's MDM system. The MDM will do a bunch of stuff with the data. However, from the point of view of you, the consumer, it will primarily send it to the billing system which will now be able to account for time of day pricing. Also, it will send those last gasp signals to the outage management system so that outage reporting will be automatic. This will make analysis and response faster and more accurate. Google appears to be leveraging their position and market power to make deals with the utilities to access that data on our behalf.

The biggest reason for smart meter initiatives is demand management. The utilities have to carry expensive excess capacity so that they can meet peak loads. If they can use interval metering coupled with better pricing and feedback systems, they may be able to change our usage patterns and smooth that load which will reduce the necessary peak capacity overhang. Also, as alternative energy sources with less predictable availability like wind power come on line the utilities will need more "load shaping" options. Ultimately they might be able to reach directly out to your smart appliances and turn them off remotely if they need to.

The laws that are mandating smart metering are focused on this demand side management. Practically speaking, most utilities will close the consumer feedback loop by offering a simple portal on the utility's web site that will let you monitor your usage in the context of your bill. However, this isn't the part of the system the utilities are excited about. The hardware and the meters are the sexy part. The contracts to build the consumer portals are probably going to go to low cost bidders who will build them exactly to low band pass requirements. In some cases they may provide provisions for customers to download historical data into a spreadsheet if they want to. A few enterprising customers will probably take advantage of this feature, but this is the hard way to do the kinds of correlations Google has in mind.

What should be apparant by now, is that the government is mandating a good idea, but they are mandating it from a utilty-centric rather than customer-centric point of view. There is naturally some overlap between utility and customer interests, but they are not identical. The utility is concerned about managing capital costs. They look at the interval data and the customer portal as a way to influence your time-of-use behaviors. They really don't care how much power you use, they just don't want your demand to be lumpy. On the other hand, we just want our bills to be low.

So, Google's initiative offers to take your data from the utility, combine it with data coming from devices in your home, and visualize it much more you-centrically. There offering will do a better job than the utility's portal illuminating structural efficiency problems in the home as well as usage pattern problems once utilities start implementing variable pricing. In short, while the utility is attempting to influence your "when I use it" decision making, Google is offering to help you make better "what I plug in" decisions along with the stuff the utility cares about.

So, what's not to like?

Google needs two distinct sources of data to make this initiative work. They need access to your data via the utility that owns your smart meter. Plus they need data from equipment manufacturers that are going to make your appliances smart or provide your home automation gadgets. It doesn't bother me at all that they get this data, as long as the utility makes it available for anyone else that might be able to innovate with it too, including me. You never know, I might want to use it for a home made gadget that sets up an electric shock on my thermostat any time my last eight averaged readings are above some arbitrary threshold, you know, just to make me think twice before turning it up.

The little bit of info that Google provides on this initiative is at their .org domain, but there is virtually no information about how to participate in data standards making, API specification, device development, or that kind of thing. If you want to participate, you pick whether you are a utility, device manufacturer, or government, fill out a form and wait for Google to get back to you. Imagine, the government fills out a form to participate in Google's initiative. Google has out governmented the government.

As I described already, governments are insisting on demand side management, but there don't appear to be any requirements to provide generic API's for meter readings or meter events. It's enterprise thinking rather than web platform thinking and we run the risk of your data being treated like utility "content." "In other news today HBO struck an exclusive deal with XYZ electric for all of their meter content, meanwhile Cinemax surprised industry watchers by locking up ABC Electric. As was reported last night, all of the remaining utility's signed with Google last week."

I'm guessing that Google is probably following the same pattern that they are using in the transit space and making (exclusive?) deals with the utilities to consume your data. You'll have to log into the utilty portal to approve their access (or check a box on your bill). But Google, or other big players that can afford to buy in, will probably be the only choice(s) you have. There is no evidence on Google.org that they are trying to create an eco-system or generalized approach that would let you, the owner of the data, share it with other value added service providers. If the utilities implement this under government mandate it will suck. If they install smart meters with stimulus package money and still don't provide eco-system API's it will worse than suck.

Any thoughts on how this plays out on the smart appliance / home automation side? Are there healthy open standards developing or is there danger of large scale exclusivity on that side of the equation too?

Google will be more innovative with this data than the electric utilities, I have no doubt about that. But I can easily imagine other companies doing interesing innovating things with my meter data as well. Especially as Google achieves utility scale themselves. If my electric utility is going to create a mechanism to share my data with companies like Google, I want them to make a generalized set of API's that will let me share it with anyone.

A quick note to policy makers in states who haven't yet finalized their programs. When you think about what to mandate, consider a more consumer-centric model (if it's easier, think of it as a voter-centric model). You should be shooting for a highly innovative and generative space where contributions and innovations can come from large and small firms alike, and where no one should be structurally locked out from participation. Don't lock us into a techno-oligarchy where two or three giant firms own our data and the possibility of innovation. If you insist on widely implemented consumer controlled API's and a less enterprise-centric model, you will not only encourage broader innovation at the consumer end, but you can use it to enhance competition on the generation side too.

Well, Google isn't really saying what they are doing, so maybe I got it wrong. Maybe they are about to get all "spectrum should be free" and roll out all kinds of draft API's specifications for comment. If you think I got it wrong, don't hesitate to let me know in the comments.

Update (2/17): Asa pointed out in the comments that Google does provide more about their intent in their comments to the California Public Utilities Commission. I missed that link before and it gives some useful hints.

Most interesting is the repeated reference to Home Area Networks (HAN). In the original post I assumed Google was taking current smart meters as a given and obtaining data from the utility MDM after it went through their data collectors. That looks like it was incorrect. Instead Google probably wants your meter to to talk to your HAN via wireless(?) and then on to them from there.

If Google can use their market position to make that data accessible off the HAN rather then from the utility MDM I think that's a good thing. Mostly because it makes possible the direct consumption and analysis of the data on my side of my home network's NAT / firewall. I didn't really touch on privacy considerations in the original post, but given that PowerMeter appears trivial from a computational point of view, I'd much rather run it locally rather than share my every light switch click with Google. If I want to know how I'm doing relative to peers I can share that data then, in appropriately summarized form.

The other point in the CPUC comments is this statement: "PowerMeter... we plan to release the technical specifications (application programming interfaces or API) so anyone can build applications from it."

This is great, but I would love to see the API's sooner rather than later. They aren't really PowerMeter API's after all, if I'm reading the situation correctly, these are proposed API's and data specifications for smart meters and smart devices. The API's that Google (and others) will be consuming, not the ones they are offering. If a whole ecosystem is going to be enabled through those API's, then the ecosystem should have a hand in developing them.

In summary, if Google manages to create a level playing field for the development of an ecosystem based on this data, I'll applaud them. Some people will use their service and, like they do with other Google services, trade privacy for targeted ads. Others will choose other approaches to using the data that provide those functions without exporting as much (or any) data.

The Kindle and the End of the End of History

This morning I was absentmindedly checking out the New York Times' bits blog coverage of the Kindle 2 launch and saw this:

“Our vision is every book, ever printed, in any language, all available in less than 60 seconds.”

It wasn't the main story for sure. It was buried in the piece like an afterthought, but it was the big news to me. It certainly falls into the category of big hairy audacious goal, and I think it's a lot more interesting than the device Bezos was there to launch (which still can't flatten a colorful maple leaf). I mean, he didn't say "every book in our inventory" or "every book in the catalogues of the major publishers that we work with." Or even, "every book that has already been digitized." He said "every book ever printed."

When I'm working I tend to write random notes to myself on 3x5 cards. Sometimes they get transcribed into Evernote, but all too often they just end up in piles. I read that quote and immediately started digging into the closest pile looking for a card I had just scribbled about an hour earlier.

I had been doing some research this morning and was reading a book published in 1915. It's long out of print, and may have only had one printing, but I know from contemporary news clippings found tucked in its pages that the author had been well known and somewhat controversial back in his day. Yet, Google had barely a hint that he ever existed. I fared even worse looking for other people referenced in the text. Frustrated, I grabbed a 3x5 card and scribbled:

"Google and the end of history... History is no longer a continuum. The pre-digital past doesn't exist, at least not unless I walk away from this computer, get all old school, and find an actual library."

My house is filled with books, it's ridiculous really. They are piled up everywhere. I buy a lot of old used books because I like to see how people lived and how they thought in other eras, and I guess I figure someday I'll find time to read them all. For me, it's often less about the facts they contain and more about peeking into alternative world views. Which is how I originally came upon the book I mentioned a moment ago.

The problem is that old books reference people and other stuff that a contemporary reader would have known immediately, but that are a mystery to me today - a mystery that needs solving if I want to understand what the author is trying to say, and to get that sense of how they saw the world. If you want to see what I mean, try reading Winston Churchill's Second World War series.

Churchill speaks conversationally about people, events, and publications that a London resident in 1950 would have been familiar with. However, without a ready reference to all that minutiae you'll have no idea what he's talking about. Unfortunately, a lot of the stuff he references is really obscure today and today's search engines are hit and miss with it - they only know what a modern wikipedia editor or some other recent writer thinks is relevant today. Google is brilliant for things that have been invented or written about in the digital age, or that made enough of a splash in their day to still get digital now, but the rest of it just doesn't exist. It's B.G. (before Google) or P.D. (pre digital) or something like that.

To cut to the chase, if you read old books you get a sense for how thin the searchable veneer of the web is on our world. The web's view of our world is temporally compressed, biased toward the recent, and even when it does look back through time to events memorable enough to have been digitally remembered, it sees them through our digital-age lens. They are being digitally remembered with our world view overlaid on top.

I posted some of these thoughts to the Radar backchannel list and Nat responded with his usual insight. He pointed out that cultural artifacts have always been divided into popular culture (on the tips of our tongues), cached culture (readily available in an encyclopedia or at the local library) and archived culture (gotta put on your researcher hat and dig, but you can find it in a research library somewhere). The implication is that it's no worse now because of the web.

I like that trichotomy, and of course Nat's right. It's not like the web is burying the archive any deeper. It's right there in the research library where it has always been. Besides, history never really operates as a continuum anyway. It's always been lumpy for a bunch of reasons. But as habit and convenience make us more and more reliant on the web, the off-the-web archive doesn't just seem hard to find, it becomes effectively invisible. In the A.G. era, the deep archive is looking more and more like those charts used by early explorers, with whole blank regions labeled "there be dragons".

So, back to Bezo's big goal... I'd love it to come true, because a comprehensive archive that is accessible in 60 seconds is an archive that is still part of history.

The Army, the Web, and the Case for Intentional Emergence

Lt. Gen. Sorenson, Army CIO, at Web 2.0 Summit

I didn't make it to the Web 2.0 Summit in San Francisco in November last year so I didn't get to see Army CIO Gen Sorenson present this Higher Order Bit talk in person. However, I thought it was cool that the Army made the agenda and luckily someone posted the video. I finally got a chance to go through it. If you didn't see the talk, or don't have the 20'ish minutes to watch it now, here's a rough summary:

- Because of security and related concerns, it takes a very long time for the Army to take advantage of new generations of technology. We tend to deploy it widely about the time it's becoming obsolete.

- However, we are now beginning to take some advantage of Web 2.0 technologies in, for example, Stryker Brigade collaboration, battle command information sharing, and command and control.

I don't think that slow technology adoption is caused by fundamental first principles, so I don't think it has to remain true. But that's a long discussion for another time. In this post I'd like to focus on Army Battle Command, Web 2.0 and Gen Sorenson's connecting the two. Specifically I'd like to talk about lost opportunity and how the same technologies can constitute a generative platform in one setting and window dressing on a temple to determinism in another.

The lost opportunity I'm thinking of isn't whether Army Battle Command is Web 2.0 enough or not. It's that enterprises tend to see web technologies as an add on to whatever they already have. Plus, they tend to focus on specific technologies rather than the combination of technology, process and policy that make a collection of technologies viable as a generative platform. "Let's add some Web 2.0 to this system; we'll use REST instead of SOAP." But the fundamental question that the web answers isn't whether REST is better than SOAP, but whether emergence is more likely to create innovation than enterprise planning, and the answer to that question is yes.

General Sorenson says in the video that "CPOF brings in Web 2.0 capability, chat, video, etc..." and then comments on "graphics, chat, use of tools..." and stuff like that to reinforce the idea that Command Post of the Future (CPOF) and the Battle Command suite it is part of has Web 2.0 attributes. Like many enterprise technologists, General Sorenson appears to be focusing on rich user experience and collaboration as the attributes that give CPOF a Web 2.0 imprimatur. While that's not unexpected, I think it leaves most of the benefits on the table and untapped.

Putting aside for the moment that CPOF isn't primarily delivered through a browser, a first step toward webness, the reality is that CPOF and other systems like it neither leverage accessible platforms nor contribute to them. It is a standalone (though distributed) computing system with gee whiz collaboration and VoIP. And while it offers some enterprise-style data services, it has none of the features of a generative platform. If I'm in the field I can't readily extend it or build on it to solve different problems, modify its proprietary underpinnings to suit my local needs, or quickly incorporate its information into other applications. If an important aspect of Web 2.0 is enabling the long tail, then this isn't Web 2.0.

I should say, this isn't a post about web 2.0 semantics. However, it's important to understand that the web's power derives from its evolution as a platform. Otherwise it's hard to see what is being missed by the military's IT enterprise (and many other large enterprises).

From the beginning the web has been generative. It wasn't CompuServe. With some basic skills you could add to it, change it, extend it, etc. Jonathan Zittrain, in his excellent book The Future of the Internet - and How to Stop It, reflects on why the Internet has experienced such explosive innovation. He argues that it's the powerful combination of user-programmable personal computers, ubiquitous networking with the IP protocol, and open platforms. Today, the emergence of open source infrastructure, ubiquitous and cheap hosting for LAMP-based sites, open API's, and the intentional harnessing of crowd wisdom has ushered in the web 2.0 era. It's an era of high-velocity low-cost idea trying that leverages the web itself as the platform for building world changing ideas and businesses.

The Internet hosts innovation like it does because it is an unconstrained complex system where complex patterns can grow out of easy to assemble simple things. Simple things are not only permitted, but they are encouraged, facilitated, and often can be funded with a credit card.

I've subscribed to the notion of Gall's Law for longer than I knew it was a law:

"A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system."

(continue reading)

Recent Posts

• The State of Transit Routing on December 15, 2008
• My Web Doesn't Like Your Enterprise, at Least While it's More Fun on November 25, 2008
• DIY Appliances on the Web? on November 18, 2008
• Apps for Democracy on November 13, 2008
• My Apple Holiday Wish on November 10, 2008
• Open Source in Defense on October 9, 2008
• Ignite Philly II on September 25, 2008
• I Am Trying To Believe (that Rock Stars aren't Dead) on September 6, 2008
• Energy Savings, Strange Attractors, ... on July 31, 2008
• The Last HOPE on July 21, 2008