Apple's XServes are extremely easy to service onsite using kits from Apple.
The service kits also make servicing the machines predictable. You can replace all the major parts in an XServe without a lot of tools or effort. Support from Apple is also pretty darned good for their server products.
As for performance issues inherent in Mac OS X, I'm sure you know more about that than I do.
True, but supercomputing code spends little time in kernel space. The bigger problem is that the G5 gives abysmal performance for anything except DSP codes compared to the Intel/AMD alternatives, due in large part to the dog slow memory architecture. Any supercomputing code that is STREAM-bound (and most are) drags on the G5.
The other thing is that when you get down and dirty, getting no-name LINUX boxes makes economic sense, even with the higher admin overhead.
Linux is one of the easiest boxes to admin in big clusters, primarily because it has been used for that purpose for so long. I'm not sure where the admin overhead is. OSX isn't bad, administratively, but some aspects of the Unix environment are not quite as usable as Linux.
You have to be able to quickly pull a bad box out, but Google doesn't seem to have a problem doing this, to great effect.
Linux has had this basic capability for ages, a few different implementations in fact, depending on what you are trying to do specifically.
Note: they're not running Linux, but a special Google OS, or so the rumors say
I can state with certainty that they are using their own version of Linux. They call their version of Linux GoogleOS sometimes, but it is just a custom Linux distro.
One of the big things is the design of their 1U, very cool and serviceable. The other is management: Apple as usual made management of a cluster very easy. In fact, they sell plug-and-play clusters, and their XGrid will even use idle time on any Mac in your organization to extend the cluster. 1,100 servers on infiniband plus 500 workstations on gigabit Ethernet makes for some serious power.