Just imagining the next generation of this type of system that could be around as soon as next year. Think of a quad eight-core Xeon with four 32-core Larrabee PCIe cards (each Xeon running one card). That’s about 8.5 TFLOPs in a 3U using off-the-shelf parts, nothing customized. You could hit a petaFLOP in nine racks.
Yes, I know technically some video cards can easily beat this, but the range of problems they can be programmed to solve is much more limited (I learned that from Folding@Home).
See link at #12.