When people are dealing with (today) a lot of music files that they have purchased, home video files from digital video recorders, and/or picture files from digital cameras, you’re talking about a lot of data and it’s very sensitive data. It can accumulate *really fast*, too...
So, *long term backup* — or your permanent *archive* — should be a real consideration, not just keeping them on your computer and some “backup” like Time Machine. I think, these days, *bare hard drives* are the best backup system that you can have, for the cost and effectiveness of the whole thing.
I spotted this article from a discussion having to do with storage and backup methodologies...
Re: Backup strategy for your SAN ?
by Eric Hansen on Jul 9, 2009 at 5:26:17 pm
Neil wrote: “With LTOs, my observation is they’re not like hard drives. You need to have the software that wrote it, just to read them. So, a Bru backup, needs Bru to restore. Retrospect needs retrospect and so on. Windows Backup (its a software built in to Windows) seems to be common across Windows systems.”
you can look back through my post history in the various COW forums and see that i have talked a lot about SDLT and LTO based archive (not backup) systems. i personally don’t believe in backing up a RAID-5 array such as Xsan. that’s why its RAID-5. have a few spare drives around to hot-swap if a drive fails. i’ve never lost media on a RAID-5 because someone was always around to hot-swap when needed. be sure to show your staff how to do this procedure.
at a former facility, we used the SDLT-600a, an ethernet-based SDLT drive. it was slow, but it worked great for archiving projects and was accessible to all computers on the office network. it used FTP and all the employees knew how to use FTP. i had researched LTO options using BRU and Retrospect with an attached LTO4 drive, but what stopped me from getting it is that it wasnt as easy as FTP (everyone needs to know how to use this system and there were definitely a few people that would never be able to learn a Retrospect or BRU-based system), and it was only available on one computer. another thing that became a huge concern was whenever the SDLT drive went down, we would lose access to our entire archive. the few times the drive broke were very critical. i would only recommend a tape-based system if you can afford 2 decks to minimize downtime. that former facility ended up getting an LTO4 system for archiving their RED footage and its been working great for them. but i know that my counterpart over there is the only one that knows the system, and personally that would drive me mad. my goal has always been to make it as easy as possible for an editor to access old footage when needed as quickly and easily as possible.
i now recommend to my clients a bare hard drive based system, where you archive your projects to 3 different drives - 2 in-house and 1 out of house that you rotate on a regular basis. a tape library should be done in the same way. hard drives are extremely cheap and getting cheaper at a faster rate than tape. then get a hard drive bay like Bob suggested above. i have been using eSATA “toasters” at a few facilities and they have been awesome. you load 2 hard drives in the top like a toaster, copy what you need, and eject.
regarding the “30 year shelf life” that Bob pointed out, i’m sure its true. but you need to keep those old decks around. tape companies advance their technology by creating new tape systems. when hard drives advance, they usually keep the same interface. look at the last 20 years and compare the different tape formats against the different hard drive interfaces. its definitely easier to hook up an old hard drive than an old tape. but you have to remember that hard drives don’t have the shelf life, so i recommend to my clients that they test their drives every 6 months and replace them when needed. thats why there are 3.
i’ve been talking about archive and not backup. for backup, i recommend that you have daily backups for your computer’s boot drive, which in many cases contains all your current projects. i would go with a cloning system like Carbon Copy Cloner (Mac) because you can get back running right away. in my experience, i have never needed an incremental backup system because no one every says to me “hey, i need a file i accidentally deleted 3 weeks ago.” its always been “crap, my hard drive died and i have to print this tape today and make FedEx.”
as for the SAN, since its RAID-5, i wouldnt worry about backing it up regularly. its more important to create a workflow for dealing with your media before loading to the SAN, and archiving after the edit is done.
e
Eric Hansen, The Audio Visual Plumber - http://www.avplumber.com/
Note these particular paragraphs...
i now recommend to my clients a bare hard drive based system, where you archive your projects to 3 different drives - 2 in-house and 1 out of house that you rotate on a regular basis. a tape library should be done in the same way. hard drives are extremely cheap and getting cheaper at a faster rate than tape. then get a hard drive bay like Bob suggested above.
[ ... ]
regarding the “30 year shelf life” that Bob pointed out, i’m sure its true. but you need to keep those old decks around. tape companies advance their technology by creating new tape systems. when hard drives advance, they usually keep the same interface. look at the last 20 years and compare the different tape formats against the different hard drive interfaces. its definitely easier to hook up an old hard drive than an old tape. but you have to remember that hard drives don’t have the shelf life, so i recommend to my clients that they test their drives every 6 months and replace them when needed. thats why there are 3.
[ ... ]
i’ve been talking about archive and not backup. for backup, i recommend that you have daily backups for your computer’s boot drive, which in many cases contains all your current projects. i would go with a cloning system like Carbon Copy Cloner (Mac) because you can get back running right away. in my experience, i have never needed an incremental backup system because no one every says to me “hey, i need a file i accidentally deleted 3 weeks ago.” its always been “crap, my hard drive died and i have to print this tape today and make FedEx.”
—
Of course, I’ve already mentioned the “SuperDuper” [Macintosh] program for super-ease-of-use for doing the job of what he refers to here... “Carbon Copy Cloner” (you could use that, too... but I prefer SuperDuper).
There was an excellent discussion on the photography forum regarding that. Triple backup is recommended, one off site in case of fire or other calamity. But they all kind of put their own twist on the exact methodology.
For now, I have the one extra hard drive, the Passports (most critical files are on two of them) and bought some very high quality dvd's (which I haven't gotten around to yet). You have to split up a lot of your files to fit on those. The Passport life span and reliability over time are rather iffy but fine in the short run. I think there are other better brands of external devices like Passports now in terms of reliability even in the short run but these were cost effective for me at the time.
Now 50 years, my stuff nobody is going to care, but I do worry about all the valuable genealogy information I put together, should have one hard copy of all that. Hard copies of my photos is not an option really, just a few albums of my best ones, will fade over time.
So your post, while complex, is timely, and I'll consider that. Glad I didn't jump right on the Time Machine, may end up with it, may not, usually take my time when I'm not sure which often pays off rather than jumping right onto something just because that's what other people are doing.
There are online storage "tanks" as well, but those can go out of business over time and subject to events out of our control. Some businesses use them, but it's not very practical for home users.
Also we need to think how much of our music and some of the photos will really matter over time and cull some of those down to the best ones.
Thank you for all that great info. I saved it so I can read it and try to comprehend it all thoroughly.
One problem with RAID 5 configurations. The chance of a sector error screwing things up on a hard drive is one in many trillion. This wasn’t a problem in the past, but it is now that hard drives have trillions of bits on them.
Basically, soon we will reach a point that if one drive on a RAID 5 fails, the odds are realistic that another drive will fail before that failed drive is rebuilt onto a replacement (RAID 5 rebuilds are slow). Going RAID 6 just alleviates the problem, and it too will be surpassed as drives get even bigger.
RAID 10 is better since the rebuild is faster and even if a second drive fails the odds are (on a large RAID 10) that it won’t be the mirror of the one you’re rebuilding. But in either case you’re throwing the dice and don’t have total reliability in case of drive failure.
The days of RAID are numbered for serious use.