My Apple Holiday Wish

I’ve been searching for a personal backup solution that doesn’t suck for, well, pretty much since I got my first computer in the 80’s, and I’m still looking.

A few years ago I was cleaning out old crap and ran across boxes and boxes of 800kb floppies labeled “1988 backup disk x.” The trash / recycling picker uppers got those along with a pile of zip disks, various CD’s, DVD’s, a USB drive or two, and a couple of bare SATA drives that I was too cheap to buy housings for. Oh, and there was even a pile of tapes in some long forgotten format in there.

After a few years of manually copying stuff to multiple USB drives, last year I was completely seduced by the “it’s like RAID but you don’t need identical drives” beauty of the Drobo. Three failures later (including one with smoke), a nasty virtual tinnitus that comes and goes as its disks transition through a perfect cabinet-resonating frequency, incompatibility problems with Time Machine and Airport Extreme, and access speeds that are too slow to serve Final Cut, and screw it. Now it mostly just sits there powered down making a Drobo-shaped dust-free spot on my desk. It’s too buzzy to listen to but too expensive to Freecycle.

Next up, Time Capsule. Still (even more) useless for Final Cut and that sort of thing, but it’s doing an ok job with backups – at least of the straight Time Machine variety. There are still a few issues though…

First off, I don’t really trust that single spinning platter. It will die some day. Plus, it’s in my house about ten feet from where my laptop is usually parked so my eggs are all in a single fire / theft / flood basket.

Apple’s Mobile Me and the Backup program that comes with it theoretically provide a solution to this issue, but unfortunately it sucks. It’s slow, much slower than a local time capsule backup because it is relying on an Internet connection. Also, it effectively requires my machine to be running all the time so that it can conduct it’s backups in the middle of the night when I won’t be competing for bandwidth or CPU cycles.

Even worse, it fails all the time. I don’t know why, but it’s finicky. A brief connectivity hiccup (or whatever) and I wake up the next day to find that my multi-hour backup died. Finally, It’s too small to be useful for more than a few key critical files. I have a few hundred gigabytes of data I’d like to secure and my mobile me account is limited to twenty.

So Apple, I don’t usually resort to begging, but here’s your chance to fix backup for me once and for all. Just update the firmware in my Time Capsule so that my fast Wi-Fi-based local backups can be incrementally streamed to either an expanded Mobile Me account or to a separate S3 account (or whatever) whenever it’s sitting at home with my network connection to itself.

I can’t leave my laptop connected for the days it would take to stream all those hundreds of Gig, but Time Capsule is just sitting there with my Internet connection doing nothing while I’m at work anyway, so give it something to do. This way I’ll have the best of both worlds, fast reasonably secure backups to my local Wi-Fi connected Time Capsule when I’m home and don’t-need-to-think-about-it remote storage that can take its time when I’m not. At the risk of way over reaching, it could even work in both directions so that if I’m on the road for an extended period, Time Machine could backup critical changes directly to Mobile Me which could then in turn incrementally stream that back to my Time Capsule.

Ok, that’s it. A simple idea I think. Can I have it by Christmas?

By the way, if the thought of all those gigabytes in your Mobile Me data centers makes you blanche (and the idea of using S3 is anathema to Apple’s do it all culture), how about a Time Capsule-based distributed hash overlay network? If every Time Capsule shipped with the option of turning on a separate partition representing about 1/3 of the disk, you could put a Planet Lab-like distributed file system in there. My files would be split into chunks, encrypted, and distributed around to other people’s Time Capsules while some of their stuff was on mine. Sort of an inverted Bit Torrent for backups, no data center required.

That would be cool but I know you won’t do it. And, from the category of “things you are even less likely to do,” if you opened up the Time Capsule firmware to third parties someone else probably would.

tags:
  • http://www.fus.net/ Matthew Fusfield

    Take a look at Mozy [www.mozy.com]; not an Apple product but I think it will do everything you want minus backup from the time capsule. I’ve been using it on both Mac and Windows for some time and it is great!

  • http://www.schrattenkalk.com/ moritz_

    What you’re looking for is wuala ( http://wua.la/ or http://wuala.com/ ). Especially since you ask for distribution and encryption. Wuala does exactly that.

  • http://brooksjordan.name Brooks Jordan

    http://jungledisk.com . . . rocks.

  • rick

    I have a Linux fileserver with separate drives for backup. A cron job that rsyncs stuff to that over wi-fi every hour and another cron that rsyncs to S3 (using s3fs). Why not Time Machine since my laptop is a Macbook? Because I don’t want to spend money for a dedicated backup storage solution when I have a perfectly good file server already.

  • http://donnyvblog.blogspot.com/ Donny V
  • davidm

    Since this is an O’Reilly blog post entitled “My Apple Holiday Wish” I was expecting something about a bedroom with Apple motif pillows, wallpaper, an Apple telephone, a huggable Steve Jobs teddy bear (minor concession with a fuzzy terrycloth black turtleneck), Apple letter head and pens/colored pencils and crayons, an Apple desk and a computer with an Apple logo on it, but I guess some of those are a given.

  • Jim Stogdill

    Thanks for the comments everyone. A few thoughts in response…

    Mozy sounds great but one of the problems I don’t want to deal with is the backup happening directly from my machine over the Internet. Too slow and consumes resources when I’m using it.

    Wuala sounds really cool in the sense that it is doing distributed hashing?? But still, fundamentally I think it still has the same issue as Mozy. The backups are using my laptops resources (for encryption / decryption) and network connection. Ditto Jungle Drive.

    Windows Home Server is pretty cool, but really, it’s doing basically the same thing as Time Capsule in the sense that the backup is still sitting there in my house; so isn’t really giving me a great DR solution.

    I think the closest thing to what I was thinking is probably the Beta version of Jungle Drive for Windows Home Server . That sounds pretty cool if it works. Use the server to make backups locally with as little impact as possible on the machine I’m working on, and then use Jungle Drive to get it offsite. Rick’s Linux file server with S3 behind it is cool and functionally the same thing but maybe not as ready for a mass market?

    The only downside of that is that I don’t really want a 100+ watt server running all the time. I don’t know what Time Capsule’s power draw is but my ideal solution is equivalent to Windows Home Server plus Jungle Drive but in a much lower power consuming package.

  • http://www.drobo.com Thomas Loverro

    Hi Jim. This is Tom Loverro, the Director of Product Marketing at Data Robotics. I am sorry to hear you’ve not had a great experience.

    You should check out the second-generation Drobo as we addressed many of the performance and acoustic complaints from our customers. The fan is much smarter and quieter and its a little less prone to drive vibration although half-height drives can still be challenging. We’ve also partnered with Western Digital and now sell Drobos bundled with WD’s 1TB GreenPower drives which we’ve found to be very quiet while still having great performance.

    I am unaware of any existing compatibility issues with Drobo and any Airport or Time Capsule products. We work closely with the folks at Apple. If you’re running the latest Airport/TimeCapsule firmware, you should be OK. I’ve also never heard a report of a Time Machine problem, so if you have found any, please email me directly and I’d be more than happy to investigate.

    You’ll also find the second-generation Drobo much faster, especially over FW800 with OS X. Final Cut can be quite demanding, so our performance is probably best for compressed streams, but I am not sure what your particular workflow is like

    Remote backup is still a challenge, but I’d love to speak with you to understand your local and offsite storage needs better so I can design future products and services to meet your needs.

    In terms of remote access, we’ve recently partnered with a startup by the name of Yoics (www.yoics.com) who has a client for PCs and Macs (in Beta) and even a client for DroboShare that doesn’t need a computer in the middle. Check out http://www.drobo.com/droboapps for more info on some of these DroboApps.

    Please drop me an email directly at tloverro [at] datarobotics.com and I’d be happy to provide you with a second generation Drobo and DroboShare to test out and just generally answer any questions you may have.

  • davidm

    Thomas Loverro, can I have one too? In terrycloth?

    I’ll use linux/backupninja/rsync, thanks.

  • http://htp://www.swordplay.tv Steven Mandzik

    Whoa – im way behind on this one. Im of the new digital generation (i think) and so back-ups for me was something I never thought about.

    I just got used to cycling through life. Every few years I would get new things and refresh, losing all the old stuff. Anything I wanted to keep I just burned to disc and transferred.

    Now im in full digital swing. I haven’t had a failure of anything yet (crosses fingers) but i do have things i need to worry about.

    So, ive got the backup external usb guy. He likes his time capsule, but I also want the external to my home back-up now.

    I didnt even know mobileme has a back-up…will have to look into that.

    Also, will have to check out the other solutions so thanx jim for posting and I hope you find an answer to put next to your standard O’Reilly steve jobs teddy :)

  • http://www.lightpole.net Doug Klein

    So Apple delivers, Time Capsule slowly uploads your backups all day long. Pretty soon all the smart consumers buy into this strategy. Your ISP starts to notice a whole boatload of upload traffic during ‘off hours’. Hmm. Running a server? Check your service contract lately? Want to see those ‘unlimited data plans’ get some pretty hard constraints applied :) ?

    Interesting to think about this in terms of reverse traffic, especially during the off peak hours. This could really screw up the current model (reminds me of the nightmare of the health club after New Years – don’t you hate it when everyone who pays for membership actually shows up? :)

  • http://ewan.to Ewan

    The idea of NAS boxes running remote backup software is pretty simple, and it’s one I’ve wanted for a long time too.

    I can only assume the reason it’s not happened yet using any of the big online backup systems is that it changes the economics of online backups.

    Currently, I backup around 20GB to Mozy Home, I don’t worry about trying to backup the rest because of the time it would take. If my Dlink DNS-323 storage box which holds around 300GB of data like movies was able to hook into Mozy Home too, I’d back everything up.

    All of a sudden, Mozy would be holding 10x the volume of data for me, and since Mozy Home is unlimited they’d still only be getting $5 a month.

  • http://www.mozy.com Devin Knighton

    Keep in mind, Jim, that after the initial backup is complete, Mozy only encrypts and backs up the changes to the files, not the entire files themselves. So Mozy runs much faster after the initial backup.

    Devin Knighton
    Mozy Inc

  • rick

    My solution certainly isn’t ready for primetime – . In many ways I’d love it if Time Machine backed up to any volume I could mount on my network, but it doesn’t do that. Mozy’s nice (I’ve got a paid license) but I can’t deal with the initial backup from my Macbook to the remote server.

    Oddly, I don’t think our requirements are that odd. As I see it, you want these things:

    1) Frequent backup to a local server. These should be fast and automatic. Set and forget basically. Mozy, Time Machine etc all do this.

    1a) Not important for me, but for some people is being able to go back N versions for a file or not deleting a file on the backup right away (even if you deleted it locally).

    2) less frequent, but automated backup from the local server to a remote server.

    The difficulty might be that many people don’t have the concept of a server on their network despite there being many small NAS-style solutions out there.

  • Jim Stogdill

    @rick Yep, that’s exactly what I want. Plus one more thing. I want it to use as little “vampire” power as possible. Ideally I’d like it to spin down into the milliwatt range (listening for new connections only) when 1) it’s done with my backups and 2) my pc isn’t connected.

    @davidm You might not believe this but your first comment was the highlight of my day. Thanks for a good laugh. And if you find out where I can get an Apple phone let me know, I always thought it would be awesome if they made one.

    @Eric You could use Jungle drive and send your giant pile O’ video to S3 but 1) it would cost you and 2) you’d have to leave your machine connected for about a week (which is why I really want this to be straight from the NAS.

  • rick

    Jim,

    The way I’ve set it up it tries every hour. If I’m out or I sleep my Macbook so that it can’t find the server, it fails and tries again next time. In my case it only copies 10-30 files so the copying is v fast.

    The power issue is the one place it falls down. My Linux box is a regular tower and I would like to find a less power hungry solution. My current thought is to find some very low power processor that supports Linux or to incorporate some wake on lan code. But hey, you have to save something for v. 2. :)

  • http://www.appliedabstractions.com Espen

    Jim,
    I use Windows Live Foldershare, which synchornizes disks (or, rather, disk directories) across the Internet. I have an old Toshiba laptop running XP in my office, with a an external 180G USB-connected hard disk. Then I have a workstation at home with a large disk. My own laptop, my wife’s, and my three daughters’ laptops are then synchronized (the My Documents directory, various other things) to these two backup/workstations.

    Foldershare can be downloaded (www.foldershare.com), has a Mac client (I supposed you could run it with just Macs). My middle daughter has a MacBook and uses Foldershare.

    Foldershare saves copies of deleted files. It allows you to access your files over the Internet, something which saved my bacon when my Lenovo’s motherboard crashed 2 minutes before I was about to give a speech (see this blog entry: http://www.espen.com/archives/2008/06/disrupted_prese.html).

    Highly recommended, and to quote Jerry Pournelle: It Just Works.

    Espen

  • http://www.appliedabstractions.com Espen

    Small addendum:

    Firstly, Foldershare does not keep a central copy of your files, but (I think) some metadata, such as file names, directory structure, etc.

    Secondly: The sync over Internet has other benefits as well: I am now in Shanghai, but my laptop is still backed up. My oldest daughter no longer lives at home, but backs up to home. And having files synched means I don’t necessarily need to bring my laptop with me: I can either work off one of the backup machines or, in a pinch, suck down the relevant file from any Internet-connected computer. Don’t think I can upload, though.

    Thirdly: The local client is very unobtrusive, you don’t notice it is there, yet the backups are fast and reliable.

    Espen (a confirmed Foldershare fan, as you probably can tell)

  • Greg Stein

    Coming in a bit late here…

    I use Mozy for my Mac laptop. It took it many days for the initial backup, but the incrementals each morning finish in just a minute or so. I bought the two year subscription, and it allows me to back up *everything* to the servers. No space limitation whatsoever.

    All that said… it has transferred bytes. Thankfully, I haven’t had to restore anything, so I can’t remark on that aspect. But the backup part is smooth and easy.

  • http://simoncpage.com Simon Page

    Personally had little to no trouble with Drobo – however as previously mentioned windows home server is pretty good and HP have a nice little device that runs this – similiar-ish to the drobo…

    http://simoncpage.co.uk/blog/2008/11/20/hp-mediasmart-server-for-your-home/

    Check it out…