Posted on 10/08/2010 8:06:52 AM PDT by ShadowAce
Hmm. This guy should give me a call. We were putting 100+ million records on Linux Slackware systems using raw devices in 1995. Have since invented something that will easily hold billions of records that can span multiple machines across any network.
I use ext4 on all of my Linux distros, if possible. It seems to me that SSDs are the next phase of the evolution of storage. Having recently purchased my first 1 TB SATA disk for my new gaming rig, I decided to run Windows 7’s system experience test on my system. I scored 7.6, 7.6, 7.7, and 7.7 respectively on each of the tests (CPU, memory, DirectX, and video), but my hard disk dumped my score down to a 5.9 (MS uses the lowest score as the final). I was shocked, to say the least, but my previous system had a 150 GB SATA disk at 10K RPM rotational speed and netted me a 6.5 on the same test.
Size truly does matter, but interface bandwidth and operating system disk operations appear to be the primary concerns.
I assume you mean a database. Most databases have a relatively small number of files on the filesystem, but the files may include millions or billions of records. That is, record =/= file.
So, are you storing each record in a separate file on the filesystem? If not, you're comparing apples and oranges.
Why wouldn’t you use a database?
http://en.wikipedia.org/wiki/ZFS
I thought that Apple almost put this into 10.5 (maybe), but had some license issues...Not really sure, but it is good to know that folks are in front of this.
It's not necessarily about storing photos or databases.
Why a database? Because in my experience trying to manage a high volume of individual data files is extremely difficult.
By manage I mean keep track of, update, backup, control access to and in general ensure the integrity of the data.
Regardless of whether you’re on an array, Windows with NTFS starts dying with only 20,000 or so files in a single folder. You’re sure to get a lock-up with half a million files.
Hmm. This guy should give me a call. We were putting 100+ million records on Linux Slackware systems using raw devices in 1995. Have since invented something that will easily hold billions of records that can span multiple machines across any network.
Yes it can. However, managing thousands of users on a single filesystem with their /home directories in a database is not very feasible, is it?
See Post #5
On the flip side, managing terabyte-sized files is even more of a pain in my experience.
“managing terabyte-sized files is even more of a pain “
Dunno, the largest I’ve dealt with was 23TB. It made restoring from the DR site a bit challenging due to bandwidth limitations but all in all it wasn’t that bad.
You can split a database into multiple files on most systems. It makes dealing with them a little easier, and usually improves performance.
This was an oracle DB hosted on SAN. We used BCVs and all that. The database itself wasn’t a problem. It was moving 23 TB between datacenters that created an issue.
Never underestimate the bandwidth of a station wagon full of backup tapes.
Hm. I tried ext4 but found it unreliable. Then again, it might have been when I migrated to Xubuntu 9.10 which, installed (once) on this system was so flaky I abandoned it for Xubuntu 9.04 and ext3. Stable.
Some flaky hardware? I dunno. Once in a while I have to pull the power plug for a few seconds and re-insert before getting the old box to boot.
Hmmm... how many files do I have on my computer...
$ sudo find / -print | wc -l
605730
That's a lot of files IMO for a simple desktop computer, but nowhere near what they are talking about.
I've dealt with directories at work with 100k+ files in them (as a result of really stupid programmers in this case), and it's not pretty when you need to do cleanup there. Thank God for xargs!
File proliferation is actually a serious issue. Desktop users, especially those that have never figured out how to properly organize files can quickly get into a state where they can't find files they are looking for, even if they know it's somewhere on the computer. Your average user doesn't know anything about how to efficiently organize directories and subdirectories. Heck, I sometimes have problems with it myself, and find myself going back through and reorganizing things periodically just to keep things straight.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.