Posted on 03/30/2013 11:56:04 AM PDT by Ernest_at_the_Beach
The maintainers of the native Linux port of the ZFS high-reliability filesystem have announced that the most recent release, version 0.6.1, is officially ready for production use.
"Over two years of use by real users has convinced us ZoL [ZFS on Linux] is ready for wide scale deployment on everything from desktops to super computers," developer Brian Behlendorf wrote in a mailing list post on Wednesday.
ZFS is a combined filesystem and volume manager for Unix-like operating systems, originally developed by Sun Microsystems for its Solaris OS in the early 2000s.
The main features that set ZFS apart from many competing filesystems are its support for astronomically large storage systems and its near-manic obsession with data integrity.
A single ZFS filesystem can theoretically scale to as many as 256 quintillion zettabytes, or 2.56 × 1041 bytes. Your Reg hack has a hard time conceptualizing just how big that is, but suffice it to say that it's big enough to hold all of the data you have ever produced.
How do we know? Because it's big enough to hold all of the data that all of humanity has ever produced, and then some. We're talking storage on a cosmic scale, using the Marvel Comics definition of the word.
But all of that storage wouldn't be much good if your data wasn't there when you went to retrieve it. All types of storage media from tapes and disks to solid-state media and beyond are subject to errors and defects. Maybe a drive fails, or the power fluctuates in just the wrong way during a write operation, or an errant cosmic ray passes by at just the wrong time; one way or another, some level of data corruption is inevitable.
On a small scale, such corruption is manageable. Disk-repair utilities, journaling filesystems, and RAID arrays do a good job of keeping even fairly large traditional storage systems airtight. But when you get into the staggeringly huge storage systems of the Big Data world, the effectiveness of such methods begins to break down, allowing silent data corruption to creep in and lurk unnoticed.
For this reason, ZFS was designed for data integrity from the ground up. It uses a complex system of checksums to spot corruption wherever it appears, and once found, errors can be corrected on the fly on a live filesystem. In addition, it supports snapshots that allow admins to roll back the filesystem to an earlier state in the case of a serious failure.
Sun released its ZFS code under an open source license as part of OpenSolaris Build 27 in 2005. Efforts to port it to Linux began soon thereafter, with the native ZFS on Linux (ZoL) port being maintained by the US Department of Defense's Lawrence Livermore National Laboratory.
So when will the latest, production-ready release of ZoL find its way into the kernel of your favorite Linux distro? Ah, there's the catch. Because of incompatible licensing and other intellectual property concerns, the ZFS filesystem module can't be distributed with the Linux kernel.
For that reason, much of the Linux community has already turned toward Btrfs, a GPL-licensed filesystem that has been under development at Oracle since 2007 and which offers similar features to ZFS.
Still, those who are impressed by the possibilities of ZFS and would like to try it out on their Linux systems or who are already using it and who want the most stable version can download source tarballs and packages from the project's website. ®
I don’t see how even this would help in the case of a physical HDD failure. When the drive goes SCREEEEECH that’s it, game over.
I’ve had ‘benign’ failures (MBR corruption) that were easily fixed with a free utility called Partition Table Doctor, but two times in the past 15 years drives went physically bad, and that was that.
From my perspective physical HDD failure is what I’m scared of, and the only option is, of course, regular backups.
Various RAID options are also suppose to minimize HDD failure consequences....as I understand it.
It's got a volume manager, so it can be configured to include redundancy to protect against HDD loss(es).
If you're worried about losing a disk and you're running Linux, you've got a couple options without zfs, either mdadm or LVM.
I don't think there's a software option included with the home user versions of Windows, but you could just get a cheap RAID1 card and do it in hardware, if your motherboard does not support it.
ZFS is really meant to be used with arrays of drives and just like a hardware RAID controllers, it generates EDAC (error-detection and correction) data and spreads it across all the drives in the array. Using the EDAC data an entire drive can be rebuilt. The penalties being about 20% extra space used by the EDAC data and it being painfully slow to rebuild a big drive.
ZFS also has a bunch of other features including live drive snapshotting. For a more of a “backup” type of feature.
But, the features are really more about “uptime” then about backing up. Just like in RAID5 or RAID6, you can have one or two drives in an array completely fail and ZFS will stay up while rebuilding the replacement drive(s). And you’re more likely to know when a drive is starting to act wonky than with a hardware RAID controller due to the extra testing that ZFS does.
All that being said, ZFS is big resource hog if you need all it’s fancy features. For example, IIRC, they recommend a minimum of 1GB of RAM for every 1TB of drive space when using all the features. So the RAM requirements are pretty hefty for a large array. And it pays to have an blindingly fast CPU with lots of cores if you ever have to rebuild a drive.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.