Twitter Approval Matrix - June 2009

by Mike Hendrickson | comments: 3

Last month I posted the first Twitter Approval Matrix with data that spanned the month of May and different sources such as Hashtag.org, scraping archives, and observations. This month I received some help from Joe Fernandez the CEO of Klout.net and Dan Zarrella the Social & Viral Marketing Scientist for danzarella.com. They provided some great 'hard' data that allowed me to better place more items on the grid this month.

A quick refresher, the matrix shows four quadrants used to describe trends found on Twitter, or related sites such as hashtag.org, tweestats.com, etc. The Y-axis is partly analytical and shows popularity (mostly through scraped numbers) or perceived popularity (in the future nominated by you). The other part of the grid is more curated and subjective. The X-axis has been plotted based on my personal opinion. You may agree or disagree with my placements and that's all good to me. After all, it is about taste. The matrix and plots do not represent a thorough analytical treatment, but rather a view of the trends that could be found in data sources allowing me to plot with some sense of relevance.

For this post, I've limited the data and activity to the month of June. Again, I'll continue with this project as long as I get enough feedback/help. So, if you are interested in contributing, you can comment here, or read the original post to figure out the best way for you to submit your plots.

I hope you enjoy this and see it as a potentially useful tool to monitor trends that your fellow readers are tracking.

Ignite Los Angeles on 7/21! Submit a Talk

by Brady Forrest | @brady | comments: 1

Ignite is coming to LA! As always speakers will get 20 slides that auto-advance every 15 seconds. We're going to be holding the geek event at Cinespace in Hollywood on 7/21. Submit a talk now.

This will be the first Ignite in Los Angeles; it is co-hosted by LA Geek Dinner. The LA G33k dinner was kind enough to let us take over their July dinner to host the first Ignite. LA G33k Dinner, founded by Heathervescent brings people with a passion for technology and the internet together over a meal where conversations happen, friendships form, and collaborations on various projects occur. L.A. Geek Dinner is an inclusive event. Find them on Facebook.

The event is free. We're hosting it at Cinespace on July 21:

6:30pm Geek Dinner starts

8pm-9:30 Ignite talks

10pm Cinespace opens to the eneral public for Dim Mak (you're welcome to stay for the band)

While the event is free, you are responsible for paying for your own food/drinks from Cinespace if you want 'em. Please RSVP to the Geek Dinner list on Upcoming.

If you're working on an interesting project, have an unusual skill, or just some interest that would be fun to share with everyone, please submit a proposal to: http://bit.ly/IgniteLA

Ignite LA is being organized by Brady Forrest, Matt Forrest, Dan Gould, and Heathervescent. If you're not familiar with Ignite check out some videos on the Ignite Show.

Four short links: 2 July 2009

by Nat Torkington | @gnat | comments: 0

1. UNESCO book: Open Educational Resources -- UNESCO's first openly licensed publication, a collection of papers and reports in the area of Open Educational Resources. (via glynmoody on Twitter)
2. ETSI 2.0 -- Paul Downey ventures into the belly of the telco beast and gives them both barrels. The whole thing is great--his talk was one of the best overviews of "how we think on the Web" I've seen. I can only imagine the sound it made as it bounced off the thick dinosaur hides of the attendees. I was reminded of the old, apocryphal quote from a Kodak executive dismissing digital cameras and their poor quality with "people love photos", when in reality it's the taking of photos that people love. Sometimes it's hard for an incumbent with large sunk costs and a vested interest in business as usual to foresee and embrace change. Indeed for a telco or large commercial software vendor the best way to predict the future is to prevent it. (via benjaminblack on Twitter)
3. Asia Pacific FTTH Market Study -- notable for Hong Kong's discovery with fibre-to-the-home customers: Uplink traffic is 3 times of downlink traffic. That link appears dead, but Google has it cached. (via previous link)
4. Shownar -- tracks blogs and Twitter plus other microblogging services, finds people talking about BBC television and radio, shows trends in appealing ways. Made by Schulze and Webb (and Dopplr's delicious Matt Jones), more detail available that you should read.

Patrick Collison Puts the Squeeze on Wikipedia

How to Cram the Wikipedia onto an 8GB iPhone

by James Turner | comments: 3

Think about Wikipedia, what some consider the most complete general survey of human knowledge we have at the moment. Now imagine squeezing it down to fit comfortably on an 8GB iPhone. Sound daunting? Well, that's just what Patrick Collison's Encylopedia iPhone application does. App Store purchasers of Collison's open source application can browse and search the full text of Wikipedia when stuck in a plane, or trapped in the middle of nowhere (or, as defined by AT&T coverage...) Collison will be presenting a talk on how he did it at OSCON, O'Reilly's Open Source Convention at the end of July, and he spent some time talking to me about it recently.

James Turner: Why don't you start by talking about your background a bit and how you got involved with working with the Wikipedia?

Patrick Collison: I guess I've always been pretty interested in Wikipedia, and I ran my own MediaWiki installations back when I was in school in Ireland. We had our own personal ones and all of the rest. Then in November of 2007, I went to visit my friend in Japan for a month. And in Japan they have all of this incredibly advanced cellular technology and all of the rest. And so because of that, they had very few wireless networks, and my phone didn't work. As a result, I actually had very little access to the Internet. I sort of realized without Wikipedia how little I really knew. And I had just got an iPhone, so I decided to try basically putting a copy of Wikipedia on the phone, so that I'd have it as I was walking around in Japan. Then basically, I spent a significant fraction of my time there in Japan, again, in 2007 writing those applications, say maybe two or three weeks, just firstly trying to decide if it was possible and putting it all together. And then it was released, I think, January of 2008.

James Turner: Now you've also worked on getting it onto the OLPC I understand. How did that occur?

Patrick Collison: I actually didn't do much of the work for this. It was actually a project led by Chris Ball who works both with FreeBSD and with the OLPC project. But I released the code to this application; it was open source from the very start. So it was pretty easy for them to take it and to port it to the OLPC. I mean there are already some applications that allowed you to put a copy of Wikipedia on your computer or something like that, but none had really been optimized for embedded or low power devices or anything like that, which obviously Wikipedia for the iPhone had to be. I think it took about two or three weeks to take the code that ran on the iPhone and then to bring it to the point where it'd run on the OLPC.

James Turner: There are obvious benefits to having Wikipedia on the OLPC, because connectivity is very important in some of those areas. So you'd want to have it local, but outside of the experience that you were just describing, isn't the point of the iPhone that you can just access Wikipedia? What are kind of the advantages of having it locally?

Patrick Collison: I actually find that you spend, or I certainly spend a surprising amount of my time without access to the internet, even with the iPhone. Say for start if you were abroad, I mean everyone knows the horror stories of the data changes AT&T will issue you with if you're roaming. But also just stuff like personally, I find that on a plane or something you have eight hours to not do much. And so I actually end up doing a lot of my Wikipedia browsing there. But even aside from connectivity issues, it actually turns out to be quite a bit faster to use the built-in, cached Wikipedia application as opposed to the website. I mean you can search in real-time with the applications. You just type a couple of characters and tap into your article, rather than firing up Safari or searching for the article in Google; then zooming in so you can tap in, et cetera, et cetera. I and most of the people I know who use the application actually end up using it even when they have internet connectivity. And maybe 20 percent of the time it's pretty useful because it's the only choice.

James Turner: Now just as a point of interest, is this an App Store app or do you have to have a jail-broken phone for it?

Patrick Collison: It was released back when only the jail-broken SDK existed. It was in that initial sort of surge of early applications. I guess the first jail-broken iPhone app, I think, happened in August, and so this was released just under six months later. And then when Apple announced the SDK, I actually originally did not intend to port it to the App store, just because I was just working on other things at the time and my company had just been bought and so it seemed like a lot of work. But then over the summer, I started getting a huge amount of email from people who had upgraded to the new version of the iPhone OS, and were now missing Wikipedia. And I started getting 20 or so emails from people per day saying they love this application and they were really missing it. Or even people saying they were continuing to use the old version of the OS just for this application. And they really hoped that I would port it so they could eventually upgrade. After receiving these emails for a while, I eventually felt too bad about not porting it. So I spent a couple of days porting it and then released it in the App Store. I wrote it and finished the port in August. And then it took about three months to wade through Apple's approval process. Around the end of October, it was released in the App Store.

(continue reading)

In Defense of Social Media (At Least Some Of It)

by Joshua-Michéle Ross | @jmichele | comments: 6

Scott Berkun just posted a great rant titled, Calling Bullshit on Social Media. I suggest everyone read it. Berkun raises good points - and I agree the hype around social media warrants taking a critical look. Despite being in general agreement, there are a few areas I can't abide, starting with this statement:

social media is a stupid term. Is there any anti-social media out there? Of course not. All media, by definition, is social in some way.

Railing against the popular lexicon is always a losing bet. Language is formed by collective agreement and it sticks because it resonates and serves a purpose. The words we use to assign to concepts can reveal quite a lot. Rather than dismissing it, we should try and learn from it. I have written before that I believe the term "social" is a new metaphor for understanding how we will transact business and conduct government. As Lakoff and Johnson so aptly pointed out in Metaphors We Live By, metaphors play a crucial role in shaping our very thought and action. We should take the "social" in social media seriously.

Next Berkun writes:

We have always had social networks. Call them families, tribes, clubs, cliques or even towns, cities and nations. You could call throwing a party or telling stories by a fire “social media tools”. If anything has happened recently it’s not the birth of social networks, it’s the popularity of digital tools for social networks, which is something different. These tools may improve how we relate to each other, but at best it will improve upon something we as a species have always done. Never forget social networks are old. The best tools will come from people who recognize, and learn from, the rich 10,000+ year history of social networks.

Well yes and no. The problem is this. Communication is the foundation of economies, government and business. When you scale up communications you change the world. It is that simple. When you radically accelerate or democratize a means of communication (I would include physical transportation in this category too) it is not a change in class (as Berkun argues) it is a change in kind.

By analogy, the railroad did not invent the wheel nor did it invent locomotion or steam power. In fact the train did not create anything particularly new. What it did was massively accelerate the ability to move people and goods across land. That acceleration changed everything… In the U.S. it standardized time, it nationalized commerce. Around the world it broke the lock of power on maritime cities that used to control commerce… and on and on.

Similarly the Internet, and social technologies in particular, do not create much that is new in the way of content (or even human interaction as Berkun notes) but the medium massively accelerates our ability to create, share, connect and collaborate. That acceleration of our innate capacity and desire to be social is exactly what makes social technologies transformative. Where I agree with Berkun's statement above is that the same rules of social etiquette will apply in this media. That is exactly what stuns so many corporations believing they can migrate essentially antisocial behaviors (hack PR blogs, social media gimmick campaigns etc.) into "social" media.

Lastly Berkun writes,

Be suspicious of technologies claimed to change the world. The problem with the world is rarely the lack of technologies, the problem is us. Look, we have trouble following brain dead simple concepts like The Golden Rule.

Agreed. People can really suck. But "change" is a value neutral term. It doesn't imply good or bad and while it is true that many negative human traits will accompany these technologies, it is hard to overstate the magnitude of the changes that are taking place as a direct result of social media - new ways to communicate, stars (including academics finding an audience) born from YouTube, bloggers redefining journalism and science, open source software dethroning traditional players, the demise of established industries like publishing, music and entertainment, with other industries like telecommunications and manufacturing, retailing queuing up for their turn. We see social technologies organizing spontaneous rallies in California, Moldavia and most recently Iran. That is change. I would also argue that the democratic promise of these tools - the promise that people can connect with each other without an intermediary (I know all of the ways that this may not turn out to be the case - but still...) holds the possibility of distributing power more evenly. If there is one root problem in much of this world - it is the concentration of power wielded by a small minority. We should celebrate any technology that lowers barriers to communication. caveat: Scott Berkun is an O'Reilly Author and in my defense, I owned his book Myths of Innovation long before I joined O'Reilly.

Four short links: 1 July 2009

Web Awards, Speed Thrills, Magazines in the Cloud, Augmented Reality

by Nat Torkington | @gnat | comments: 0

1. The Onyas -- New Zealand web design awards launch, from the people behind Webstock and Full Code Press. The name comes from "good on ya", the highest praise that traditionally taciturn New Zealanders are allowed by law to give.
2. The Year of Business Metrics: Don't make your users run away! -- wrapup of the Velocity conference. AOL: Users who had a slower experience view far fewer pages. Some interesting notes on performance from a Google-Bing study: Notice that as the delays get longer the Time To Click increases at a more extreme rate (1000ms increases by 1900ms). The theory is that the user gets distracted and unengaged in the page. In other words, they've lost the user's full attention and have to get it back. [...] As much as five weeks later, some users, especially those who saw delays greater than 400MS, were still searching less than before. (via timoreilly on Twitter)
3. Printcasting -- very simple content management system for print magazines that lets anyone start a magazine, add content, sign up contributors, sell ads, and go. Clever!
4. Pachube Augmented Reality Hack -- sexy hack that pushes all my buttons: computer vision, Arduino, sensor network, ubiquitous computing, pervasive alternate reality cyborg villians with chalk designs hellbent on world domination and the enslavement of the human race to use as meatsack AA batteries for their sex toys. Okay, four out of five ain't bad. (via bruces on Twitter)

Pachube Augmented Reality Demo

Velocity and the Bottom Line

by Steve Souders | comments: 0

Velocity 2009 took place last week in San Jose, with Jesse Robbins and I serving as co-chairs. Back in November 2008, while we were planning Velocity, I said I wanted to highlight "best practices in performance and operations that improve the user experience as well as the company's bottom line." Much of my work focuses on the how of improving performance - tips developers use to create even faster web sites. What's been missing is the why. Why is it important for companies to focus on performance?

That question was answered at Velocity last week by speakers from AOL, Google, Microsoft, and Shopzilla.

• Eric Schurman (Bing) and Jake Brutlag (Google Search) co-presented results from latency experiments conducted independently on each site. Bing found that a 2 second slowdown changed queries/user by -1.8% and revenue/user by -4.3%. Google Search found that a 400 millisecond delay resulted in a -0.59% change in searches/user. What's more, even after the delay was removed, these users still had -0.21% fewer searches, indicating that a slower user experience affects long term behavior. (video, slides)
  
• Dave Artz from AOL presented several performance suggestions. He concluded with statistics that show page views drop off as page load times increase. Users in the top decile of page load times view ~7.5 pages/visit. This drops to ~6 pages/visit in the 3rd decile, and bottoms out at ~5 pages/visit for users with the slowest page load times. (slides)
• Marissa Mayer shared several performance case studies from Google. One experiment increased the number of search results per page from 10 to 30, with a corresponding increase in page load times from 400 milliseconds to 900 milliseconds. This resulted in a 25% dropoff in first result page searches. Adding the checkout icon (a shopping cart) to search results made the page 2% slower with a corresponding 2% drop in searches/user. (Watch the video to see the clever workaround they found.) Image optimizations in Google Maps made the page 2-3x faster, with significant increase in user interaction with the site. (video, slides)
• Phil Dixon, from Shopzilla, had the most takeaway statistics about the impact of performance on the bottom line. A year-long performance redesign resulted in a 5 second speed up (from ~7 seconds to ~2 seconds). This resulted in a 25% increase in page views, a 7-12% increase in revenue, and a 50% reduction in hardware. This last point shows the win-win of performance improvements, increasing revenue while driving down operating costs. (video, slides)
  

These case studies provide real world numbers that show the benefits of making your site faster. Other Velocity sessions share techniques for implementing performance improvements, including sessions from me, Doug Crockford, and the Facebook and Google frontend teams. But what about the user experience? In his session, Matt Mullenweg (of WordPress fame) makes sure we remember the importance of how the user feels while interacting with our site:

That's why [performance] is important and why we should be obsessed and not be discouraged when it doesn't change the funnel. My theory here is when an interface is faster, you feel good. And ultimately what that comes down to is you feel in control. The web app isn't controlling me, I'm controlling it. Ultimately that feeling of control translates to happiness in everyone. In order to increase the happiness in the world, we all have to keep working on this.

Thanks to the Velocity speakers & their organizations for overcoming the many challenges required to present this data for the first time. We're now equipped with the financial justification, the technical know-how, and the visceral motivation to go out and make the Web a faster place. We'll have more performance success stories next year. Your company could be one of them! Capture your performance improvements and bottom line impact. We'd love to hear from you at Velocity 2010.

Everyblock's Code is Open-Sourced

by Brady Forrest | @brady | comments: 2

The code for Adrian Holovaty's Everyblock has been released. The open-sourcing of the site's system were apart of the Knight News Challenge Program. Everyblock is a very impressive site that aggregates and geocodes local data -- news, crime, fire, restaraunt inspections and reviews - and then lets users define their interests down to the block-level.

Adrian made the announcement on 6/30. Here's the list of newly open-sourced, GPL'd goodies found on Everyblock's new Code page:

The main package (probably the thing you're looking for) is the publishing system, known as ebpub.
Second, the packages ebdata and ebgeo contain Python modules for processing data and making maps.
Third, the packages ebinternal and everyblock round out the code that powers EveryBlock.com. They're internal tools and are likely not of general use, but we're including them to be complete.
Finally, ebblog and ebwiki are our blog and wiki software, respectively. Because, dammit, the world needs another Django-powered blogging tool.

Django fans, Python geohackers and anyone who wants to build a local data aggregator are going to be thrilled. Adrian was one of the co-creators of Django and was one of the first Google Maps Mashup creators.

Everyblock has only launched in major US cities. There's plenty of room in the market for locals to create their own version. Everyblock spends a lot of time curating the incoming data feeds so I doubt that anyone will be able to roll out new sites too quickly. One thing to note: the trademark Everyblock is not available. However, the Everyblock team would not mind being acknowledged if you use their code. Personally I get a lot of value of Everyblock in my city. I get a daily email with all the crime, news and errata near my house.

Everyblock is now going to move onto the second stage of its existence. About five months Adrian blogged about the dilemna they would be facing when they open-sourced their software. As he said at the time:

But now we've reached an interesting point in our project's growth: our grant ends on June 30, and, under the terms of our grant, we're open-sourcing the EveryBlock publishing system so that anybody will be able to take the code to create similar sites. That's a Good Thing, in that EveryBlock's philosophies and tools will have the opportunity to spread around the world much faster than we could have done on our own, but it puts the six of us EveryBlockers in an odd spot. How do we sustain our project if our code is free to the world?

At the time I suggested that they try to federate with new everyblocks. After yesterday's announcement I mailed Adrian to ask him for a hint about their future plans, but for now he's keeping mum.

The Hacker Ethic - Harming Developers?

by Jim Stogdill | @jstogdill | comments: 22

On Monday Neil McAllister posed the question "is the hacker ethic harming American developers?" Slashdot picked it up and Tim forwarded it to the Radar list. As you might expect, it resulted in some spirited discussion.

James Turner kicked things off with this response (it has been slightly edited from its email form). After James lays out his argument I'll reply with my thoughts. Then we hope to hear from you. Let us know what you think.

I've worked in a lot of organizations that thought that the kind of rigid deforesting paradigms that Nayar is referring to were the magic bullet to keeping all three of the variable (dollars, time, features) under control. Without exception, all they did was get in the way and reduce the productivity of the most senior people to the level of the most junior. All of them exhibited some degree of failure, some catastrophic.

The India shops *love* methodologies like UML and the like, specifically because the problems have been reduced to a simplistic enough granularity that they can be doled out to junior-level staff, who may have only been onboard a few weeks because of the massive churn over there.

At least 3 times at 3 different companies, I've seen major pieces of work brought back in-house because the Bangalore team had fallen so far behind or proved so unable to get beyond the literal description of work that they were endangering the project. When you combine the time difference with a tendency to halt dead in their tracks as soon as they hit a stumbling block, it can be a recipe for disaster.

There are certainly some good Indian shops, and I know some outstanding Indian developers (most of whom have come to the States.) But I find Nayar's comments hilarious. It's akin to someone saying that American football players aren't employable in Jamaica because they aren't able to limbo well. Look at the most successful Web 2.0 companies today, most of them started as garage enterprises with a few talented developers, not a 60 person team of UML jockies following some Arthur Anderson project management program. Heck, look at Google Labs.

In huge projects, you obviously need some master planning and coordination to make sure the tracks meet at the right place to drive in the spike, but I don't see any effort being made these days to right-size the amount of project overhead to the needs of the projects. Instead we get a one-size-fits-all approach that smothers anything but the largest project in paperwork. Even some of the original authors of the Agile manifesto, when I've talked to them, point out that part of being Agile is picking and choosing the right components of project management that make sense for a given task.

Nayar's remarks are incredibly self-serving. "We're the best, because we can mindlessly follow some arbitrary and flawed development process." Or is he claiming that Indian projects do better QA? Not in my experience...

This entire debacle is representative of a problem I think is endemic in the industry these days, the inability or unwillingness to engage in rapid prototyping. Every successful project I've ever worked on (and I've worked on some fairly large enterprise-sized projects), we started by designing and coding a quick "throw-away" skeleton of the application, that let us look at how the thing worked, where the unseen warts were, and where the vendors had lied about their APIs, etc. This is the crucial and neglected stage in project design, one that most modern design paradigms ignore or actively discourage. Even Agile tries to jump in and start coding the finished application from the get go, although if project teams were willing to aggressively refactor (a tenant of Agile), early project work could be a rapid prototype (although the story model of scrum really doesn't fit well with this, unless you make the prototype a story...)

This is also something I've never seen an offshored team do particularly well...

Well, I'll be damned if I'm going to jump in and take the side that says hacking is bad for American programmers. First off, I don't need that kind of flame bait and second, I don't believe it. I think approachable programming is hugely important because that's how many people get into the field in the first place. However, my reaction to the article was very different than James' and I might as well try to explain.

I'm not going to take an opposing position, but it's not really an orthogonal position either. Maybe it has a power factor of about .7 or so. Here's my (also edited) response...

When I when I read McAllister's piece, at least some of it resonated with me. Before we were bought by a large firm, we were a small company that grew from nothing to 250 people, about 200 or more of them were programmers. So, a whole lot of my time over three years was spent hiring programmers and building cohesive teams that could deliver to our customers.

In our hiring we aggressively hired hackers into the mix. We wanted outside-our-industry thinking and we thought they brought in creativity. We called it "hiring weird, but not weird weird." Occasionally we pushed it to weird and a half. For our efforts we got creative problem solving and interesting (but frequently weird) dinnertime conversation when we travelled.

However, our pollyanna idea of "disciplined teams catalyzed with a bit of weird" didn't always work out.

That leads to the bit that resonated with me: the sense that hacker = distilled essence of American individualism combined with lots of ISTJ Myer's Brigg's Type Indicator. Individualism is a trait that I hold dearly, but it can make a cohesive team effort difficult if people are unwilling to suborn themselves to the goals of the team. Remember those tee shirts the football team always wore at your university? "There is no I in team?" I sometimes joked that I was going to make a batch that said "I'm the Me in team."

Maybe we were just growing fast and it was going to take more storming and norming than I had patience for, but at times it was a struggle to get everyone to see past their individual biases and focus on what we were trying to achieve, and we couldn't do what we were trying to do with teams of one.

But it really wasn't a hacker problem if hacker means self taught like McAllister implies. We had a lot of people with CS degrees and we used to talk a lot about whether and how their degrees had prepared them for their jobs.

Separate from the individualist approach to development, few of our recent graduates came to us prepared with the terminology and practices of any development approach (or engineering approaches like continuous integration etc.). They knew how to code, but not how teams coded.

At one point I gave a talk on agile software development to about 100 CS students at a university in Philadelphia and I asked them to raise their hands if they had ever done a team project with greater than two people on the team. I don't recall anyone raising a hand. Then I asked if they had ever covered development methodologies in their classes and a few acknowledged they had, but it had been abstract classroom stuff only. That part surprised me.

I'm not sure that the "sanctity of engineering" argument really makes that much difference. I have little faith in McAllister's scheme to do computer engineering instead of computer science coursework.

My undergraduate degree was in Mechanical Engineering and I can only imagine how useless I would have been to a firm that actually did engineering, and for mostly the same reasons. I knew how to take integrals and I still know the packing ratio of a hexagonal close packed material but I didn't know squat about how a complex machine actually got designed in a team setting. It's interesting to note that the Engineer in Training exam I took (a precursor to the professional engineer's exam) didn't probe my knowledge of team practices at all.

Maybe there just isn't time in an undergraduate degree to teach everything that an engineer needs to know. Plus, can you imagine the drop out rate in CS/CE if ITIL was a required course?

Since James mentioned Google I'll switch gears and muse about ecosystems for a moment... I guess I tend to bristle when I hear that everyone should just develop software the way Google does.

Google is to computing what LA was to Aerospace and Electronics in the early 60's. It's gravitational force attracts five sigma talent (probably a bunch of six too) in ways that the rest of us can only envy. More generally, Silicon Valley has had programmer talent flowing into it for the last twenty years the way Hollywood sucks pretty people out of the midwest.

Maybe it's not as obvious because you can't spot brains the way you can spot an oddly beautiful wait staff, but the valley has been the vortex of a talent-laden embarrassment of riches for a long time and, if you work there, you might not even notice it. However, I think that at some level this effects what kinds of processes work when you build software. I think it's at least a little part of the reason why an ERP system in a manufacturing town gets implemented differently than MapReduce (there are other reasons too having to do with software as product vs software as supporting infrastructure). Combine that with the very clear shared vision of "lets do something great and get rich together" thing that valley firms often have, and well, it's easy to see how smart people coalesce to build amazing stuff.

It's easy to denigrate Arthur Anderson's progeny or the offshore firms they compete with, but they do different work, with a different talent pool, for different ends, and with a very different set of personal and organizational incentives. Or, put another way, Kelly Johnson didn't build the SR-71 with General Motor's engineers, and General Motors didn't design the Chevy Cavalier with the Skunkworks' processes. However, even at the Skunkworks, Johnson's brilliant engineers did conform to a process and work together as a team toward a shared vision. And, conversely, I bet a lot of talent is left on the table at General Motors because of processes too restrictive in their attempt to remove all uncertainty.

So,... maybe it's possible that Google's (or the valley's in general) processes are appropriate to an ecosystem that, because of the intellectual environment and potential for riches, is rich in IQ and initiative. So it ends up feeling more "special forces" and less like "infantry regiment." And over there closer to the hump in the normal distribution curve, or in a different cultural environment outside of the valley, a different flavor of processes may be effective?

The counter argument to that, which I'll go ahead and provide, is that I once helped teach a team of engineers at midwestern defense contractor how to do agile development. The effect was amazing and immediate and their productivity and satisfaction went up tremendously; until their management freaked out and shut it down when they "perceived" that it created too much uncertainty in their processes.

Well, it's obvious that I don't know the answers here, so, with that, I'll stop thinking out loud.

What do you think?

The US Online Job Market Was (still) Down Big In June 2009

by Ben Lorica | @dliman | comments: 0

Updating my post from early June, the U.S. online job market^† still hasn't shown signs of recovering from steady declines that began in September of last year. Compared to the same period last year, there were 50% fewer job postings in June 2009.

An alternate view highlights the start of the downward trend, as well as the smaller than expected seasonal bounce from Dec-08 to Jan/Feb 2009. In a normal year, the number of postings decline in December (as employers table job searches for after the holidays) and recovers sharply the following Jan/Feb. While job postings did bounce back in Jan/Feb 2009, the seasonal bump was less than half of what occurred in previous years.

No geographic region has been exempt from the downturn in online job postings. There have been sharp declines in all states, ranging from -59% in DE, WY, and MN, to -38% in MD, OK, VA.

In closing, we still haven't detected the green shoots that some forecasters have been crowing about over the last few months. If one were to take an optimistic perspective, the worse year-over-year decline occurred in April. OTOH, we are still staring at a 50% decline in June 2009. So while we may have hit the bottom in April, we need a few strong(er) months before we can comfortably announce the arrival of green shoots.

(†) In partnership with SimplyHired and Greenplum, we maintain a data warehouse that contains most U.S. online job postings dating back to mid-2005. Data for this post was through 6/28/2009.

Radical Transparency: The New Federal IT Dashboard

by Tim O'Reilly | @timoreilly | comments: 14

Today, at the Personal Democracy Forum in New York, Vivek Kundra, the US national CIO, unveiled the new IT spending dashboards at usaspending.gov. The dashboards are designed to help Vivek and the CIOs of individual government agencies get a handle on the effectiveness of government IT spending.

At the top level, the dashboards provide a view of spending by major government department, with graphs showing performance against schedule, costs, and the CIO's assessment of how well they are meeting their objectives. For example, here's a stark view of IT performance at the Veteran's Administration (click to expand image):

49% of the VA's IT projects are behind schedule, and Roger Baker, the agency CIO, deems that a full 63% of the projects are in need of serious attention. (Here's a recent article that outlines Baker's tough-love plans for IT at the Veteran's Administration.)

As you drill down, you get to dashboards for individual IT projects (800 projects and approximately $20 billion in budgeted spending). Each project shows the responsible government official, the prime contractors on the project, the CIO's evaluation of its progress against goals, and each month, an update showing an update of that progress. (We'll show one of these later in this article.)

The dashboards are an incredibly ambitious undertaking. In the first place, there has never been a government-wide view like this of all IT spending, and the progress of projects. What's even more remarkable, though, is that the dashboards are being shared with the public. It's a bit like having your performance review posted on the company bulletin board for all to see.

In notes provided to press in advance of the announcement, Vivek Kundra wrote (italics mine):

Over the past several years, we have witnessed numerous public failures of major information technology systems and just last year saw roughly one third of all investments reported as poorly planned or poorly performing. Many of these investments may never deliver on their original promises. With over $75 Billion in annual federal information technology spending, we need a new foundation for management - one built on the values of transparency, accountability, and responsibility....

Data is powerful. It enables monitoring, reporting, and meaningful analysis that leads to better decisions. Yet, in the case of federal information technology, we lack insight into project performance. Poor data quality coupled with infrequent reporting has led to lack of meaningful analysis and bad decisions. Numerous failures and cost overruns may have been avoided with timely access to accurate information.

The Administration is committed to using technology to move past these barriers. In the IT Dashboard, the public has a platform for unprecedented access to useful, unfiltered data regarding the performance of IT investments. Information available includes responsible government officials and contractors as well as project performance data, updated monthly. This enables better decision-making, giving us the ability to turn around poorly performing projects and to divest from those which no longer make sense.

In making this data publicly available, we are providing unfettered access to investment performance to its true owners - the American people.

Vivek explained that last point further in a telephone conversation with me last night. I asked him about the level of buy-in across the government for this kind of radical transparency about the performance of projects. He said:

"It's a cultural transformation, in terms of recognizing that we are in the public square. The work that we do is work that is supposed to be performed in the interest of the American taxpayers. And so making visible how we're performing means fleshing out these complicated issues in the public square. Culturally, making the shift is much better than letting it hide under the veil of secrecy.

(continue reading)

Four short links: 30 June 2009

Military Open Source, Social Govwork, Dietbot, and US IT Dashboard

by Nat Torkington | @gnat | comments: 0

1. Military Open Source Software Conference -- 12-13 August 2009 in Atlanta.
2. Govloop -- a "Social Network for Gov 2.0". Gov 2.0 could easily become the intersection of talk radio and social media consultant inanity. As with the Web 2.0 lunacy, when everyone who could spell wiki tried to sell one, you should cultivate the art of identifying and sidestepping the bozos, the time-wasters, and the charlatans who use buzzwords as a convenient alternative to thought. (via cheeky_geeky on Twitter)
3. Introducing the Autom -- a personal robot to help you lose weight. Developed by Initiative Automata as an offshoot from MIT researcher Cory Kidd, Autom has conversations that encourage you to record your diet and exercise. The theory is that the added benefit of interaction will help you stick with the diet longer, increasing the chance that it will stick. Trials showed Autom users stick with their "weight loss regimen" twice as long as pencil-and-paper. (via So, Where's My Robot?)
4. USA Government IT Dashboard Launches -- Vivek Kundra's latest project, a dashboard giving insight into government spending. Contractors, CIOs, projects, schedules, and data via an API. Built in Drupal!

Recent Posts

• Bing's Sanaz Ahari on System Feedback (2 of 2) | by Brady Forrest on June 30, 2009
• Personal Democracy Forum conference: initial themes | by Andy Oram on June 29, 2009
• Want A Job? Learn SharePoint, Says Gary Blatt | by James Turner on June 29, 2009
• Bing's Sanaz Ahari on Query Level Categorization (1 of 2) | by Brady Forrest on June 29, 2009
• Four short links: 29 June 2009 | by Nat Torkington on June 29, 2009
• Silicon Valley's First Phone Company - A conversation with Ted Griggs | by Joshua-Michéle Ross on June 27, 2009
• How Active is Twitter Now? Tweespeed | by Brady Forrest on June 26, 2009
• Scott Berkun on Why You Should Speak (at Ignite) | by Brady Forrest on June 26, 2009
• Four short links: 26 June 2009 | by Nat Torkington on June 26, 2009
• Apple, the Boomer Tablet and the Matrix | by Mark Sigal on June 25, 2009
• Naming an Emerging Movement | by John Geraci on June 25, 2009
• Four short links: 25 June 2009 | by Nat Torkington on June 25, 2009