I plan to do a complete write-up sometime later. Everything is now clustered into three distinct groups (frontend, backend, database.) All machines are dual processor. Frontend now runs on 933 MHz and 1.4 GHz machines, backend runs on three 2.8 GHz machines, database runs on 933 MHz and 1.4 GHz machines. We have a dual 450 MHz machine for misc. tasks (it's incapable of any serious work these days.) We've upgraded to Giga-Ethernet for our internal network, the forward facing network is constrained to 10 Mbits/sec (a point-to-point connection to Verio, our co-location host.)
My goal is to memcache everything, to hit the database for as little as possible. Right now the two frontends have 2 GB of memory each, with a 1 GB memcache on each. The backends have 1 GB of memory each, with a piddly 256 MB memcache on each. The database machines have 3 GB of memory all to themselves.
The only disks that matter are the DB, they're 15K RPM 74 GB SCSIs for the data, one on each server. I have another 15K RPM 36 GB SCSI for the O/S and log files and some of the tables (just to balance load.)
Oh, and this guy (PDF) is my hero.
The system is capable of syncing across the Internet. In fact, I sync the database over a SSH forwarded port, for backup and development. However, we rely on a master write database, which means any child node would need to communicate with the master database to store anything. Site redundancy isn't yet an option. I can address that with a complete rewrite of the database and software, including some of the concepts we know and love, which is something I both look forward to doing and dread.
That's about what I figured. I could do the hardware but would have issues on the bandwidth.
Right now I have a half a dozen dual Xeon's with a few GB each, 2 dual Opterons with a few GB each, and a number of single CPU machines - P4 and Alpha mostly, and my trust Sparc. A few TB of disk, internal GigE network. I may have to pick up another pair of dual or even quad Opterons soon for a new project.
If you ever get to the point of looking at a live backup site, feel free to send me email. At some point I'm going to have to upgrade my connectivity.
Nice to have the hardware, but it's the power bill that sucks.
-tb