The iPhone: Tricorder Version 1.0?

The iPhone, in addition to revolutionizing how people thought about mobile phone user interfaces, also was one of the first devices to offer a suite of sensors measuring everything from the visual environment to position to acceleration, all in a package that could fit in your shirt pocket.

On December 3rd, O'Reilly will be offering a one-day online edition of the Where 2.0 conference, focusing on the iPhone sensors, and what you can do with them. Alasdair Allan (the University of Exeter and Babilim Light Industries) and Jeffrey Powers (Occipital) will be among the speakers, and I recently spoke with each of them about how the iPhone has evolved as a sensing platform and the new and interesting things being done with the device.

Occipital is probably best known for Red Laser, the iPhone scanning application that lets you point the camera at a UPC code and get shopping information about the product. With recent iPhone OS releases, applications can now overlay data on top of a real time camera display, which has led to the new augmented reality applications. But according to Powers, the ability to process the camera data is still not fully supported, which has left Red Laser in a bit of a limbo state. "What happened with the most recent update is that the APIs for changing the way the camera screen looks were opened up pretty much completely. So you can customize it to make it look any way you want. You can also programmatically engage photo capture, which is something you couldn't do before either. You could only send the UI up and the user would have to use the normal built-in iPhone UI to capture. So you can do this programmatic data capturing, and you can process those images that come in. But as it turns out, at the same time, shortly after 3.1, the method that a lot of people were using to get the raw data while it was streaming in became a blacklisted function for the review team. So we've actually had a lot of trouble as of late getting technology updates through the App Store because the function we're using is now on a blacklist. Whereas it wasn't on a blacklist for the last year."

Powers is hopeful that the next release of the OS will bring official support for the API calls that Red Laser uses, based on the fact that the App Store screeners aren't taking down existing apps that use the banned APIs. Issues with the iPhone camera sensors pose more of a problem for him. "In terms of science, it's definitely a really bad sensor, especially if you look at the older iPhone sensor, because it has what's called a rolling shutter. A rolling shutter means that as you press capture or rather as the camera is capturing video frames or as you capture a frame, the camera then begins to take an image. And it takes a finite number of milliseconds, maybe 50 or so, before it is actually exposed to the entire frame and stored that off into a sensor. Because it's doing something that's more like a serial data transfer instead of this all at once parallel capture of the entire frame, what that causes is weird tearing and odd effects like that. For photography, as long as it's not too dramatic, it's not a huge deal. For vision processing, it's a huge deal because it breaks a lot of assumptions that we typically make about the camera. That has gotten better in the 3GS camera, but it's still not perfect. It is getting better, especially when the camera's turned on the video mode."

(continue reading)

The Minds Behind Some of the Most Addictive Games Around

If you've wasted half your life playing Peggle, Bejeweled, Zuma or Plants vs. Zombies, blame these guys!

The gaming industry tends to focus on the high end products, first person shooters that crank out a bazillion polygons a seconds and RPGs which spend more time developing the plot in cut scenes than in actual gameplay. But for every person playing Borderlands, there are scores playing casual games like Bejeweled and Zuma. PopCap Games has been at the forefront of casual game development, with a catalog that includes bestselling titles like Peggle and Plants vs Zombies, in addition to the two previously mentioned. I recently had a chance to talk to Jason Kapalka, one of the founders and the creative director of PopCap. We discussed the evolution of PopCap, how the casual gaming industry differs from mainstream gaming, and the challenges of creating games that can be engaging, without being frustrating.

James Turner: Could you start by talking a little bit about your background and how you came to PopCap and what you did before then?

Jason Kapalka: My career in computer games started back in the early '90s, when I was writing for the magazine, Computer Gaming World, doing various reviews and articles. In '95, one of the editors from the magazine left to join an internet dotcom start-up in San Francisco called TEN, the Total Entertainment Network. He invited me to come down there and work there, which I did. And TEN evolved over the dotcom boom and bust cycle, from a very hardcore gaming service into what eventually turned into Pogo.com around 1999. I worked there initially on hardcore games. One day, I was working on Total Annihilation tournaments, and then the next day, someone said, "Hey, design bingo." And I was sort of like, "Oh. Bingo? Okay."

That was the beginning of my casual game design career, I guess. And yes, I was there at Pogo. I helped design a lot of the structure for their casual games until around 2000 when I left, and Pogo eventually went on to get bought by Electronic Arts, of course. I left in 2000 and started PopCap with two other guys, Brian Fiete and John Vechey who are these guys from Indiana that I'd met earlier, around '97. They had made an internet action game called ARC that we'd produced on TEN, and we stayed in touch. In 2000, we all thought we wanted to try something different. So we all left our respective companies to start PopCap. As you might remember, 2000 was not the best year for internet companies. So we didn't really realize that the entire industry was collapsing. We had an interesting time initially. Luckily, our ignorance protected us, I guess.

PopCap started from there, just the three of us working out of our apartments. And over time, we'd say, "Well, I guess we need to hire an artist." And I'd say, "Well, I guess we need to hire maybe another guy here to program this stuff." And then eventually, maybe someone should look at the books or whatever, so we'll hire someone to take care of the bookkeeping. And it kept going like that until eventually we thought that maybe we needed an office. And from there, suddenly, we've got nearly 300 employees now in 2009. So it's been an interesting kind of experience. We never really intended PopCap to get anywhere near as big as it has today.

James Turner: How would you describe PopCap's place in the market today?

Jason Kapalka: I guess it's a bit odd. Casual game companies exist in these strange spaces where they're often the developer and the publisher at the same time. And then they also publish stuff with other guys, where they're sort of rivals, but also they're partners. There's a lot of this co-opetition thing going on. PopCap is obviously a developer, and we develop a lot of games. We used to publish other people's games. And we still do indirectly. in that we have SpinTop Games. which is a company we bought a couple of years back. They distribute a lot of other people's games through their site. But primarily, I think we develop and then publish titles. But we primarily focus on publishing our own titles. So we're kind of a self-publisher, I suppose.

James Turner: That's actually something I wanted to ask you about because one of your distribution channels now is Steam, which is another company's portal for their games and others. How do you see that relationship?

Jason Kapalka: Steam's been really good. We work with lots of different portals. Steam is one of many that our typical game would go out on. On Steam, on Real Arcade, Big Fish Games, Yahoo Games, MSN, WildTangent, a whole bunch of smaller channels. So Steam was just one of several. It's been interesting in that it was developed differently than a lot of those other ones. Steam is definitely much more of a hardcore game distribution channel than something like Real Arcade. So initially, when we started on Steam, it was uncertain whether our games were going to really fit in. Initially, a lot of the ones we tried on Steam didn't really work too well for their audience. Hidden object games don't do especially well with Steam users, for example.

The turning point for Steam was probably when we did Peggle Extreme with Valve. I don't know if you remember that. Peggle had just come out, and the guys at Valve really liked it. We were talking and we had some weird ideas. Someone had the odd suggestion to do sort of a miniature-themed version of Peggle that featured all of the Orange Box's characters, the Half-Life, MT Team Fortress guys. It was a really strange idea, because that was a fairly mature violent kind of franchise. And certainly, it didn't seem like the obvious fit for Peggle. But, on the other hand, we thought, "Well, what the heck? We can try it and it's only going to go on Steam anyway so it's not like it'll offend the soccer moms necessarily." So we tried that out, and it went up. And we were all kinds of nervous because we didn't know -- it had launched initially as a free download with the Orange Box. And even though it didn't cost people anything, we were still kind of wondering if there was going to be this big backlash from the hardcore community about, "What the hell is this cheap little pinball thing doing in the middle of my Orange Box product."

But actually, the response was really good. I mean, the Orange Box guys all really liked Peggle a lot. And ultimately, that led them to go and seek out and buy the regular versions of Peggle which made Peggle suddenly this fairly big success on Steam. Which a month or two ago, before that, didn't seem very likely that this game with unicorns and rainbows would be selling well on Steam. So after that, that sort of seemed to kind of be -- it sort of opened the floodgates a little bit. And now a variety of our games do very well on Steam. Obviously, Plants Vs Zombies was the last one that had quite a hit there. Not everything. There's still some of our games that are clearly more casual and that don't particularly work well on Steam. But the ones that do work there seem to really work well.

James Turner: There seems to be a fairly different expectation level for casual games in terms of graphics and such. Do you think that's a natural result of how they're produced and what they're intended for? Or could you see something like Plants Vs Zombies but with the graphics levels of a Half-Life?

Jason Kapalka: It's certainly possible. I mean in some cases, we're not intentionally trying to make the games low fidelity. We try to do the best art direction we can. Although the usual contradiction, or decision to be made, there is we also want to make games as accessible as possible. So we want Plants Vs Zombies to play on every crummy netbook and seven-year-old computer your mom has and all of these types of things. And so that tends to mean that we try to work and have good art, but usually make the technical requirements very modest. We've been working at making things that can scale well so that on a good computer, you'll get a really nice experience and it'll still scale down to play on a lower-end computer. But that can be challenging in itself. So usually, we err on the side of not worrying about the graphics being too high-end because our experience is showing that a good game with not very fancy graphics can sell very well, like Plants Vs Zombies. And I think that game has good graphics, but it's definitely limited. It's only got 800X600 resolution and so forth. But on the other hand, we've seen plenty of games in the casual space that have really good graphics and they sell very poorly if they're not a fun game. So accessibility and fun definitely, for us, end up being a first priority over graphics. And especially 3-D or technically impressive graphics versus just good art direction.

James Turner: You would think Nethack and Rogue would be the ultimate proof that you can have good game play without good graphics.

Jason Kapalka: Sure, I love Roguelike games. We have lots of Nethack fans over at PopCap, which seems a bit weird in that they're obviously not very casual in many regards. But yeah, they're good exemplars of that principle that graphics are not as important as game play.

(continue reading)

Why Google and Bing's Twitter Announcement is Big News

Tweets will finally become first class web citizens

Lurking innocently on Google's blog this afternoon, like many of their big announcements, was the bombshell that they have reached an agreement with Twitter to make all tweets searchable. This followed an earlier announcement at the Web 2.0 conference by Microsoft that Bing has also arranged to make tweets searchable.

This is not only a huge thing for Twitter, it is also well past due. Until now, Twitter really hasn't been a first class web citizen, because you're not really part of Web 2.0 until you're searchable by Google (and, I suppose, Bing). Sure, you can read someone's tweets from Twitter, or get a thread via a #tag, but the full text searching capabilities that make things really usable on the web, largely powered by Google, have been missing.

Making tweets searchable is a major usability improvement as well. Twitter handles are cute, but sometimes obscure as well. Perhaps people will start using more full names in their tweets in addition to @ references, which would let you find tweets about people without having to know what their handle happened to be.

It appears that Twitter is going out of their way not to play favorites in the search space, by cutting deals with both Microsoft and Google. Microsoft seems to be ahead of the game right now, since they have a live site up, whereas the announcement from Marissa Mayer of Google only hints at things to come over the next few months.

The Bing interface is interesting, it seems to be a hybrid of a web search engine and a twitter search. Typing in a term gets you back both the latest tweets that match the keywords, as well as web pages that more than one tweet share in common that also match the keywords. This is a tacit acknowledgement that a lot of the useful content of Twitter is found in the web pages that are linked from the Tweets.

If I had to guess, I'd say that Tweets will show up more traditionally on Google, as just another kind of search result, that can be narrowed in the same way that you can narrow results to just images or movies. I guess we'll have to wait and see on that.

Life With TED - Micromanaging Your Carbon Footprint

I've spent three days watching my power consumption like a hawk, here's how it's going

I've been interested in having a better handle on my electrical consumption for a long time. Our family regularly goes through 1100-1200 kWh a month, and it's been frustrating that I couldn't really get a grip on where or when the power was really being used. I want to get my power usage under control for three reasons:

1. I want to reduce my $180-a-month-and-climbing power bill. Public Service Company of NH (PSNH) has one of the higher electricity rates in the country (we have a nuke we're still paying off, among other things.)


2. I'm seriously investigating adding solar to the mix, now that a 30% federal tax credit, a $6,000 state rebate, and lower prices for the panels have converged. It would be great to get my usage down into the 600-800 kWh average output I've been told I can expect a month from a system, and zero out my PSNH bill on a yearly basis.


3. I'm a firm believer in reducing carbon emissions, I'd like my 14 year old son to have a world to grow up in. I've already cut my fuel oil use in half (to a still awful 250 gallons a month in the winter, but it's a huge house...) Cutting my electricity is the next low-hanging piece of fruit on the tree.

I had been tracking Google PowerMeter, a Google initiative that lets people monitor their energy usage online, but it was only available to customers of electric providers who were using so-called "Smart Meters". Smart Meters send usage data back to the provider, and PSNH isn't one of them.

Then, this week, Google announced on their blog that normal mortals could now order a device called The Energy Detective (or TED, as he's known by his friends...) TED is made by Energy, Inc. out of South Carolina, and consists of a minimum of two components. The first piece is an inductive current measuring device that lives out in your circuit breaker box. The second is a gateway device that plugs into a wall socket and has an Ethernet jack. Optionally, you can also get a stand-alone display, so that you don't need a computer to view your usage.

Wiring the sensor device into your box is fairly straightforward. You clamp the two sensors around the mains as they come into the box. You also have to wire the device to the two "hot" phases of your 220V service (which requires two free breakers in your box on different phases), and a third wire running to neutral. If you have some basic electrical savvy, you can do it yourself, but I decided to wimp out, since my box is so crowded (after-effects of having a transfer switch put in for a generator...), so I shelled out the $85 to have an electrician put it in.

The gateway unit communicates with the sensor unit via signals sent over the house AC. As with anything using the power lines to communicate, I found the unit was very particularly to which outlet I plugged it into. It really doesn't like to share a circuit with a computer, for example. Neither of the two plugs which was actually next to a network hub would pick up a signal, but one in an adjacent room that happened to have a network jack did.

Once you have the gateway talking to the sensors and plugged into the network (it uses DHCP to get an address), you can surf to it using any browser. I can even get to it using Safari on my iPhone. The "home" screen is a dashboard, showing various statistics about current demand and your daily, weekly and monthly averages. You can view the data in terms of kWh, dollars (once you tell TED how much you pay for power, it can even handle peak period and tiered pricing models), or pounds of CO2.

All of the ranges on the dials and bar-graphs are configurable, so if you want 3kWh to be "red", you can set it up that way. You can also configure refresh rates. Clicking on the "Graphing" tab lets you view your usages second to second, minute by minute, or by daily or weekly aggregates.

It's these graphs that I have found to be most useful. You can start to see all sorts of interesting patterns, like the "heartbeat" of my furnace turning on and off at night, when the rest of the house is otherwise quiet.

I can also see the huge hump when my son wakes up in the morning, and proceeds to turn on every first floor light in the house. I was even able to tell that my wife had turned on the dishwasher before she left for school one morning.

(continue reading)

David Hoover's Top 5 Tips for Apprentices

Finding a Good Mentor is Key

If you're a senior developer with years of experience under your belt, it may be hard to remember what it was like coming out of college with a newly minted CS degree, and entering the workplace. But as David Hoover argues, helping these newcomers to the workforce to succeed can be the difference between effective, motivated developers and confused, discouraged ones. Hoover is the author of the new O'Reilly book Apprenticeship Patterns, and he says that people coming right out of college may, in fact, be less motivated than someone who has been working for a while. "One of my theories is computer science education is really hard, and it's expensive. And so when you're done with it, you're ready to cash in and sit back for a little while. 'Hey, I just spent a lot of money. I spent a ton of time and effort and pain on four years of getting this certificate and okay, now it's time to make that pay off.' You're definitely going to be less incentivized to start a new job, and now realize that you've got so much more to learn still. As opposed to someone who's just coming up, who's going to be at a big disadvantage knowledge-wise, but is probably actually going to be at a big advantage motivation-wise because they're going to be hungry, and just assume that they have to learn everything on their own. Whereas, like I said, some computer science people are going to be disincentivized. They're going to be surprised that they've come into their first job and, geez, they have to learn source control and they have to learn unit testing and they have to learn about these different processes that we use. And some programs prepare you for that stuff; some programs are very theoretical and very outdated. And you just have a ton to learn in your first gig."

According to Hoover, one way to ease the transition into real life development is to use an apprenticeship model. His book draws on his own experience moving from being a psychologist to a developer, and the lessons he's learned running an apprenticeship program at a company called Obtiva. "We have an apprenticeship program that takes in fairly newcomers to software development, and we have a fairly loose, fairly unstructured program that gets them up to speed pretty quickly. And we try to find people that are high-potential, low credential people, that are passionate and excited about software development and that works out pretty well."

Hoover says that most developers have benefited from one or two key people in their career that helped them move along. "For people that had had successful careers, they only point back to one or two people that mentored them for a certain amount of time, a significant amount of time, a month, two months, a year in their careers." He also points out that finding that person may mean looking outside your company. "For me personally, I wasn't able to find a mentor at my company. I was in a company that didn't really have that many people who were actually passionate about technology and that was hard for me. So what I did is I went to a user group, a local Agile user group or you could go to a Ruby user group or a .net user group, whatever it is and find people that are passionate about it and have been doing it for a long time. I've heard several instances of people seeking out to be mentored by the leader, for me that was the case. One of our perspective apprentices right now was mentored by the leader of a local Ruby user group. And that doesn't necessarily mean you're working for the person, but you're seeking them out and maybe you're just, "Hey, can you have lunch with me every week or breakfast with me every other week." Even maybe just talking, maybe not even pairing. But just getting exposure to people that have been far on the path ahead of you, to just glean off their insights."

(continue reading)

How NPR is Embracing Open Source and Open APIs

Daniel Jacobson Will Talk About the NPR Open API at OSCON

News providers, like most content providers, are interested in having their content seen by as many people as possible. But unlike many news organizations, whose primary concern may be monetizing their content, National Public Radio is interested in turning it into a resource for people to use in new and novel ways as well. Daniel Jacobson is in charge making that content available to developers and end users in a wide variety of formats, and has been doing so using an Open API that NPR developed specifically for that purpose. Daniel will talk about how the project is going at OSCON, the O'Reilly Open Source Convention. Here's a preview of what he'll be talking about.

James Turner: Can you start by explaining what NPR Digital Media is and what your role with it involves?

Daniel Jacobson: Sure. NPR is a radio organization, of course, and the Digital Media Group, of which I'm a part, handles, essentially as I describe it, everything that is publishable by NPR that does not go to a radio. So that includes the website, podcasts, API, mobile sites, HD radios, anything that has some sort of visual component to it. So Digital Media as a group is responsible for producing that content, producing all of those distribution channels, managing all of those relationships.

James Turner: And what is your particular role there?

Daniel Jacobson: I manage the application development team that is responsible for all the functional aspects of all of the systems, which includes our CMS, all of the templating engines for the website, for the API, for the podcasts, all of the engines that drive that.

James Turner: Now NPR is an organization that consists of a lot of member stations kind of flying in close formation. What's your relationship with the content producers? To what extent do they have their own stuff, and to what extent do you work together?

Daniel Jacobson: Those member stations are really exactly that; they are members of NPR. They essentially buy NPR programming. They're distinct organizations from us. NPR is a content producer and distributor. They buy our programming and broadcast it out to the world. They also have their own corresponding web teams that can take NPR content and also produce their own content and create their own websites. So in the Digital Media Team, we take a lot of pride and effort in providing services that help those member stations better serve their communities and their listeners and audiences, using NPR content and using their own content. We work with them to try and satisfy their missions. And to the extent that they need NPR services or content, we work hard to try and provide those. The API is one massive step, I think, in making it much easier for them to do what they need to do without a whole lot of intervention from us, where previously they would have to pull in content in much more arduous ways. So the API, I think, is a step in the right direction to make it more of a self-service model.

James Turner: Since you've mentioned the API, that's what you're going to be talking about at OSCON. We've already talked to the New York Times and the way they're opening up their content through APIs. What are you doing with yours?

Daniel Jacobson: Well, we launched ours formally at OSCON last year. And at that time, we essentially opened up our entire archive. So anything that you can get on npr.org is available through the API, to the extent that we have the rights to distribute it. There are some rights restrictions, for example, for receiving photos or stories from sources that we have not cleared rights to redistribute. Those are getting suppressed through a rights filtering engine on our API. Everything else that you can get on npr.org, you can get through the API. That includes full text. It includes images, audio, video, everything like that. Throughout the last year, we have added more features. We included the layer of "mix your own podcast", for example, which allows people to not only get the content in audio form, but also to download it as a podcast-type item. And all of that is available through search terms or totally customized queries. So what the API really does is it enables people to take the content, make widgets, or do whatever they want with essentially everything that is on npr.org and get to audiences that we are not getting to.

(continue reading)

Making Government Transparent Using R

Danese Cooper thinks it will be an important tool in Open Gov

With Open Source now considered an accepted part of the software industry, some people are starting to wonder if we can't bring the same degree of openness and innovation into government. Danese Cooper, who is actively involved in the open source community through her work with the Open Source Initiative and Apache, as well as working as an R wonk for Revolution Computing, would love to see the government become more open. Part of that openness is being able to access and interpret the mass of data that the government collects, something Cooper thinks R would be a great tool for. She'll be talking about R and Open Government at OSCON, the O'Reilly Open Source Convention.

James Turner: Why don't you start by describing where you came from, and you're involved in, and what your interests are?

Danese Cooper: Okay. I'm Danese Cooper. I serve on the board of the Open Source Initiative. I have been serving for the last eight years. And I'm also currently employed by Revolution Computing, which is a start-up focusing on an open source language called R, as in the letter R, that is very useful for analytics and statistical analysis. I'm also an Apache member. And I also serve on an advisory board for Mozilla.

James Turner: One of the two panels you're going to be speaking on at OSCON is on open source and open government. If you could talk a little bit about what interests you about open government and also what open government means to you.

Danese Cooper: Sure. Well, along with a lot of open source people, I got interested in the Obama campaign and in helping President Obama get elected. And part of why he was so compelling was that the vision of how Washington needed to change is pretty close to the way that we think about working collaboratively in open source. The night that he was elected, there was a great little clip on CNET of a Republican commentator actually explaining open source as exactly what I just said. It was a really brilliant little two-minute clip. He pointed at The Cathedral and the Bazaar, that canonical document about how open source works. And he said, "Microsoft is the cathedral. It's their way or the highway. And the bazaar is a bunch of people working together grassroots to collaboratively build the things that they need. And so Obama's basically asking for the government to become open source, and the problem is Washington isn't really like that right now."

So anyway, that's the transformation that has to happen in order for government to really be transparent. To me, open source government is transparent government. There's been an awful lot of shenanigans in recent political history, like the last decade has been pretty crazy in terms of things happening that couldn't be traced back to any source. Even just the way we vote and the way that voting is managed, and the fact that the software that runs the machines that we vote on is not open source so it can't be inspected. And nobody knows quite what it does. There are all of these stories of weird updates to the software that happened right before major elections in states where there are strange results. Transparency, in the same way that it helped the software industry transform, could really help the government transform. So that's what I'm talking about. There's a bunch of other people on that panel. My good friend, Brian Behlendorf, and I co-proposed it. And he's actually taken the next step. He helped found Apache. And he's run off to Washington to work on projects that are interesting to the Obama government to try to figure out how to help them to more open source solutions. And he'll be talking about his progress on that panel. So I think it's a pretty exciting panel.

(continue reading)

Sequencing a Genome a Week

Radar Talks to OSCON Speaker David Dooling

The Human Genome Project took 13 years to fully sequence a single human's genetic information. At Washington University's Genome Center, they can now do one in a week. But when you're generating that much data, just keeping track of it can become a major challenge in itself. David Dooling is in charge of managing the massive output of the Center's herd of gene sequencing machines, and making it available to researchers inside the Center and around the world. He'll be speaking at OSCON, the O'Reilly Open Source Convention. His talk, titled The Freedom to Cure Cancer: Open Source Software in Genomics, will be about how he uses open source tools to keep things under control, and he agreed to talk about how the field of genomics is evolving.

James Turner: Can you start by describing what it is you do and how you came to be doing it?

David Dooling: Sure. I work at the Genome Center at Washington University in St. Louis. We are one of the handful or so of large scale genome sequencing centers around the world. What that means is essentially we participate in large genome sequencing projects that some people may have heard of, like the Human Genome Project, Thousand Genomes Project, things like that. And involved in that is a lot of data processing, laboratory processing, tracking and all sorts of things, so it's a rather large enterprise.

There are about 300 or so people that work here. And how I came to work here was that about eight years ago, I decided that I wanted to get more into programming and more into open science. So I took a job as a programmer here at the Genome Center and gradually worked my way around to where I am now, where I oversee all of the software development and IT infrastructure here at the Genome Center. And it's a fairly large IT infrastructure.

We have somewhere around three petabytes of storage online, and somewhere north of 3,000 cores in our computational cluster. And we're generating terabytes, tens of terabytes of data, per day with our current sequencing instruments. The sorts of things that we're doing now as we transition from more fundamental evolutionary types of projects, such as the Human Genome Project and subsequent projects like the Mouse Genome Project, we've done things like corn and things of that nature, now we're doing more and more sequencing projects related to medicine and medical sequencing.

Last year, we published the full cancer genome sequence. In doing both the cancer and the normal, we were able to determine the differences between those two genomes and begin to identify what might've possibly caused cancer in that individual. So projects like that. We're also doing projects with metabolic syndromes, like diabetes, and several other cancer projects as well. That's essentially what we're doing and how we're doing it and how I got here.

James Turner: Genomics is an area that seems to be on the steep part of the hockey stick curve right now. In just a decade, we've gone from sequencing one genome over a period of years to doing them routinely. Can you talk a bit about what's enabled this acceleration?

David Dooling: Well, a whole host of things. But I think really at the core was the changing fundamentals of sequencing itself. For a long time, DNA sequencing was based on a process invented by Sanger, sometimes called Sanger Sequencing, sometimes called capillary electrophoresis now because of the last revision of the instruments that were generated. But essentially with that approach, you did reactions in 96 plate wells. You processed sequence in these 96 plate well chunks. And you did reactions in there. You loaded them on the readers, and the readers read out sequence for each of those 96 wells. So that's sort of how you processed it. And at the height of that sort of sequencing, which was only a few years ago, we had about 130 or so of those instruments each churning about 15 to 20 runs per day. Each run gave you 100 pieces of sequences. You had 100 or so machines. And so you got on the order of a few thousand sequence reads, that's what we called them, because of the way the instrument read the information.

Now, since that time, 454 was first [of the new generation of sequencers] and then Solexa came, which was later bought-out by Illumina, and the ABI SOLiD has a platform. There's one from Helicos as well. And then several other third generation, those first being the second generation, sequencers have come out. And what those do is greatly increase the parallelism with which you're able to process DNA and sequence it. So instead of a few thousand runs per day, or a few thousand reads per day, you may get a few million reads per run. And these runs, for some of the platforms, do take a little bit longer. But the parallelism of it increases your throughput tremendously. And so now we have about 35 to 40 of these highly parallel instruments in-house. And with that, we're able to sequence the human genome to complete coverage in less than a week.

So the main driver has been this change in the sequencing technology and the parallelism of it. It's a fundamentally different chemistry, different physics. The flipside of it is that we talked about the hockey stick, and so that hockey stick is the sequencing hockey stick, but it's brought several other hockey sticks along with it, mainly the amount of data that these things generate. And the amount of processing power that is required to process that data has increased greatly as well. Much faster than Moore's Law over the last two years or so. Whereas with those original instruments, you would generate on the order of megabytes per day, now we're doing tens of terabytes per day with these new instruments. And then processing that, instead of taking a single processor a few minutes, it can take a small cluster a few days to actually analyze the data from each of these runs.

Those are the main things. The enabling technology was the change in the sequencing chemistry itself. And then what had to come along with that was building these infrastructures to be able to track these things and process these things and store all of this data as the instruments increased in their abilities.

(continue reading)

Open Source is Infiltrating the Enterprise

Forrester's Jeffrey Hammond Says There's Plenty of it Around, if You Look

There's a persistent perception that open source software is being ignored in the enterprise, that IT management fears it and it ends up being more costly to deploy than proprietary solutions. That's certainly the perception that some major software vendors would like you to have. But it's Jeffrey Hammond's job to dispel those perceptions, at least when they aren't accurate. As an analyst for Forrester Research, Hammond covers the world of software development as well as Web 2.0 and rich internet applications, so he sees how open source is being used on a daily basis. He'll be speaking at OSCON, the O'Reilly Open Source Convention, talking about the true cost of using open source, and he gave us a sample of what's going on in the enterprise at the moment.

James Turner: To begin, can you just talk about the areas you cover for Forrester?

Jeffrey Hammond: Sure. I'm in the Application Development and Program Management Group at Forrester, which means I write for folks like developers, architects and development managers. And open source is a little bit of an interesting topic for me because it kind of crosses roles. But what we tend to find is that a lot of time it's developers and maybe development managers that are pulling open source into the enterprise. And that, I think, is why I have it as a research area. But in addition to open source, I also cover Web 2.0. I cover rich internet application development. I cover software change and configuration management and application life cycle management, software modeling, mobile development, IDEs and programming languages. So a pretty wide-variety of development related stuff.

James Turner: There's obviously a lot of apocrypha and maybe even some downright misinformation about how open source is being used in the enterprise. Can you illuminate things a bit?

Jeffrey Hammond: Sure. You know, it crosses a wide-variety of spectrums. But anyone who tells you that it's not being used strategically for mission critical applications isn't talking to the right people, because I see it used that way as well as on individual projects at the departmental level. I'll give you one example: I have talked with a large organization that's using multiple open source products and projects and frameworks as the core of it's reservation systems, and they're pushing 30,000 transactions a second through what's essentially an open source infrastructure. So I defy anybody to say that that's not mission critical. And it certainly contributes revenue to their business. I've also talked to organizations that are almost completely open source in their development environment. So all of the software code that they write is being built with open source tools. I think all you have to do is look at the latest data from the Eclipse Community Survey, which they published about three weeks ago, and it's pretty hard to make the argument that open source adoption is not pervasive and accelerating in the market.

James Turner: What are some of the real success stories you've seen with open source in commercial settings?

Jeffrey Hammond: Well, I just mentioned the one about the travel and transport provider, but I've seen open source used as the basis for in-store sales systems that are PCI compliant. So that would counter another fallacy that you often hear; you can't build secure software with open source. And I defy you to find an example of a system which would need a higher level of governance and qualification than something that's handling credit card data. And, yet, we see organizations successfully deploying software with things like OpenSolaris and networking stacks that are based on open source software. I've also seen it used in financial services organizations and by a major airline in Europe, which is using open source at the core of their operating system strategy. And they're running their SAP installations on top of an open source operating system framework and saving a million dollars simply by doing that.

So the uses are many and varied, but generally, the goal is to save money. That's where organizations tend to start. And then what tends to happen is the more that they become comfortable with using open source, and the more that they apply it successfully, the more they start to realize that there are benefits other than cost savings that they can take advantage of. And that's when you start to see them turn from open source opportunists into open source advocates. It's interesting to watch that transformation happen over a year or a two-year period at a large company. They start looking for opportunities to replace commercial products, things at the app server level, things at the business intelligence level, things in the web content management space. And all of these are opportunities where there are real, credible, open source projects that are used by large organizations successfully.

(continue reading)

Patrick Collison Puts the Squeeze on Wikipedia

How to Cram the Wikipedia onto an 8GB iPhone

Think about Wikipedia, what some consider the most complete general survey of human knowledge we have at the moment. Now imagine squeezing it down to fit comfortably on an 8GB iPhone. Sound daunting? Well, that's just what Patrick Collison's Encylopedia iPhone application does. App Store purchasers of Collison's open source application can browse and search the full text of Wikipedia when stuck in a plane, or trapped in the middle of nowhere (or, as defined by AT&T coverage...) Collison will be presenting a talk on how he did it at OSCON, O'Reilly's Open Source Convention at the end of July, and he spent some time talking to me about it recently.

James Turner: Why don't you start by talking about your background a bit and how you got involved with working with the Wikipedia?

Patrick Collison: I guess I've always been pretty interested in Wikipedia, and I ran my own MediaWiki installations back when I was in school in Ireland. We had our own personal ones and all of the rest. Then in November of 2007, I went to visit my friend in Japan for a month. And in Japan they have all of this incredibly advanced cellular technology and all of the rest. And so because of that, they had very few wireless networks, and my phone didn't work. As a result, I actually had very little access to the Internet. I sort of realized without Wikipedia how little I really knew. And I had just got an iPhone, so I decided to try basically putting a copy of Wikipedia on the phone, so that I'd have it as I was walking around in Japan. Then basically, I spent a significant fraction of my time there in Japan, again, in 2007 writing those applications, say maybe two or three weeks, just firstly trying to decide if it was possible and putting it all together. And then it was released, I think, January of 2008.

James Turner: Now you've also worked on getting it onto the OLPC I understand. How did that occur?

Patrick Collison: I actually didn't do much of the work for this. It was actually a project led by Chris Ball who works both with FreeBSD and with the OLPC project. But I released the code to this application; it was open source from the very start. So it was pretty easy for them to take it and to port it to the OLPC. I mean there are already some applications that allowed you to put a copy of Wikipedia on your computer or something like that, but none had really been optimized for embedded or low power devices or anything like that, which obviously Wikipedia for the iPhone had to be. I think it took about two or three weeks to take the code that ran on the iPhone and then to bring it to the point where it'd run on the OLPC.

James Turner: There are obvious benefits to having Wikipedia on the OLPC, because connectivity is very important in some of those areas. So you'd want to have it local, but outside of the experience that you were just describing, isn't the point of the iPhone that you can just access Wikipedia? What are kind of the advantages of having it locally?

Patrick Collison: I actually find that you spend, or I certainly spend a surprising amount of my time without access to the internet, even with the iPhone. Say for start if you were abroad, I mean everyone knows the horror stories of the data changes AT&T will issue you with if you're roaming. But also just stuff like personally, I find that on a plane or something you have eight hours to not do much. And so I actually end up doing a lot of my Wikipedia browsing there. But even aside from connectivity issues, it actually turns out to be quite a bit faster to use the built-in, cached Wikipedia application as opposed to the website. I mean you can search in real-time with the applications. You just type a couple of characters and tap into your article, rather than firing up Safari or searching for the article in Google; then zooming in so you can tap in, et cetera, et cetera. I and most of the people I know who use the application actually end up using it even when they have internet connectivity. And maybe 20 percent of the time it's pretty useful because it's the only choice.

James Turner: Now just as a point of interest, is this an App Store app or do you have to have a jail-broken phone for it?

Patrick Collison: It was released back when only the jail-broken SDK existed. It was in that initial sort of surge of early applications. I guess the first jail-broken iPhone app, I think, happened in August, and so this was released just under six months later. And then when Apple announced the SDK, I actually originally did not intend to port it to the App store, just because I was just working on other things at the time and my company had just been bought and so it seemed like a lot of work. But then over the summer, I started getting a huge amount of email from people who had upgraded to the new version of the iPhone OS, and were now missing Wikipedia. And I started getting 20 or so emails from people per day saying they love this application and they were really missing it. Or even people saying they were continuing to use the old version of the OS just for this application. And they really hoped that I would port it so they could eventually upgrade. After receiving these emails for a while, I eventually felt too bad about not porting it. So I spent a couple of days porting it and then released it in the App Store. I wrote it and finished the port in August. And then it took about three months to wade through Apple's approval process. Around the end of October, it was released in the App Store.

(continue reading)

Recent Posts

• Want A Job? Learn SharePoint, Says Gary Blatt on June 29, 2009
• Walking the Censorship Tightrope with Google's Marissa Mayer on June 16, 2009
• John Viega Talks About Beautiful Security on June 10, 2009
• Google Squared is an Exponential Improvement in Search on June 3, 2009
• Velocity Preview - The Greatest Good for the Greatest Number at Microsoft on May 18, 2009
• Google Engineering Explains Microformat Support in Searches on May 12, 2009
• Velocity Preview - Keeping Twitter Tweeting on May 7, 2009
• Tim O'Reilly - Why Twitter Matters for News on May 7, 2009
• Where 2.0 Preview - DARPA's TIGR Project Helps Platoons Stay Alive on April 21, 2009
• Where 2.0 Preview - Building the SENSEable City on April 16, 2009