With ten years in HPC under my belt, I can vouch for that. Linux flat-out is the rule in HPC.
I suppose on the top stuff they roll there own.
Guess it is based on BSD:
The following distributions match your criteria:
1. PelicanHPC GNU Linux
PelicanHPC is a Debian-based live CD image with a goal to make it simple to set up a high performance computing cluster. The front-end node (either a real computer or a virtual machine) boots from the CD image. The compute nodes boot by Pre-Execution Environment (PXE), using the front-end node as the server. All of the nodes of the cluster get their file systems from the same CD image, so it is guaranteed that all nodes run the same software. The CD image is created by running a single script, which makes it possible to customise the live CD image with extra Debian packages.
2. Rocks Cluster Distribution
Rocks is a complete "cluster on a CD" solution for x86 and x86_64 Red Hat Linux clusters. Building a Rocks cluster does not require any experience in clustering, yet a cluster architect will find a flexible and programmatic way to redesign the entire software stack just below the surface (appropriately hidden from the majority of users). Although Rocks includes the tools expected from any clustering software stack (PBS, Maui, GM support, Ganglia, etc), it is unique in its simplicity of installation.