I barely know how to turn my computer on, but might a better title to this post be: Can you structure a problem that can be finished faster on disk than in-memory? It pays to be specific.
Given that nearly all operating systems use virtual memory, all bets are off anyway.
Makes sense. Instead of putting the string together in memory and writing it to disk when it’s completed, you are writing to the disk as it’s assembled. So you’re skipping a step.
Interesting comments at article. Many saying that the code was poorly written.
Cheating. If the objective is to build something on the disk, building it on the disk is going to be faster, duh.
I can’t believe how retarded this is.
It makes no sense to concatenate the string in memory and then write it to disk, since in either case you will be writing the string sequentially to disk, anyway. Java, Python, C#, and other "managed" languages will always do this more slowly because their strings are immutable, which any decent coder knows.
Best approach: find out the allocation block size of a file on disk, pre-allocate one buffer of that size, memory write to that buffer, flushing the whole block to disk when it's full; this avoid the penalty of zillions of memory allocations and garbage collections and writes a block of optimal size.
In most cases, just pre-allocating a moderately sized block of memory without knowing the best block size is good enough and may even be preferable, because the underlying OS is going to optimally block IO, and probably also cache that at a secondary level.
The key point is to avoid over-allocating managed objects, and again, most good coders know to do this, even if people writing stupid research papers don't...
The real issue isn't memory vs. disk, its what the language you are using does to perform the string concatenation operation.
The fastest technique will be one that does string concatenation in memory while the disk write of the previous string section is completing, so that the disk latencies are used for string building. Oh, and of course the string concatenation code should be designed to run in cache and avoid any virtual memory paging or extra memory copy operations.
The key to performance is understanding how the system works, and writing code at a low enough level to be able to control how it interacts with the system. That's why C and C++ still get used.
The technique of "writing 1 byte at a time" to the disk is really just a way of utilizing the buffering present in the I/O system to queue up disk writes. All the interesting stuff is actually happening in memory, however its being done by clever system code written by people who understand how to get high performance.
A well written version of the string concatenation test should be able to write data to the disk as fast as the disk can write data.
I smell a bug.
That’s the way it was written, the test. It saw the flaw of some nature and then wrote a perfectly good set of conflicting code. We called them bugs and the people who exploit them hackers.
I’ve spent many of nights watching and analyzing processor bus activity on a logic analyzer along with a profiling running program in the OS to believe that they just didn’t find a bug to exploit.
Once you replace you mechanical hard drive with an SSD, you will then know what fast really is.
I bought a Samsung 850EVO and wow!
I’m never going back.
So cache is not necessarily king?
Yeah, right. Like that's gonna happen. The monkeys churning out code today probably think Big Endian and Little Endian is a children's book about Native Americans.
doing the operation in memory then doing a single 1m write to disk is still FAR faster then 1m 1 byte writes followed by 100k of 10bytes, etc.
even if the memory version was written as a single byte at a time, it would equate to the 1m 1 byte writes. the other writes would be slower
Generally, you should set a buffer size of 4-16K and format your app's output directly into the buffer, if possible. You may wish to use multiple 4-16K buffers, so that you are writing into the current buffer while one of your past buffers is being transferred to disk asynchronously. When you fill the current buffer, it should be queued for output, and you should switch your output-formatting activity to scribble on a previous buffer which has already been written. When you are done, you should remember to queue your final buffer for output and wait until all buffers have been written. Then please close the file.
The optimal buffer size and number of buffers should be determined by experiment.
One million single writes to disk can be a much different proposition if the test has the disk all to itself than it is on a busy system where every write operation can potentially have to get queued and wait for some other process to release the disk channel.
IMHO
I'm sure it is possible to construct very narrowly tailored circumstances where what they are describing makes sense, but it's such an artificial construct that it's not really useful. It's simply a reminder to never use the word 'never'.
It only proves that you can design a test to do stupid things that don't really apply in the real world.
First and foremost, is the fact that memory is everything. In order for a process to write to disk, it must first put that data in a buffer, which is (gasp) MEMORY. In most modern, enterprise level systems, there is a ton of cache (more memory) sitting in the disk subsystem to receive the data from the operating system prior to it being written to disk.
Let's see them run an application or database doing real-world work and see how their theory holds up. I got $100 that says "not very well"
In simple terms the test was to get a string of bits written to the disk in a given order
So a one step operation—write to the disk— is faster then a two step operation—organize the bits in memory—then write to the disk....
Gee that a shock...(/sarcasm off)
Anyone who uses a buffered database knows this is nonsense.