One of SGI's really big mistakes was coming out with a line of x86 Windows NT workstations. Although they were faster than PCs with the same CPU, they were a financial disaster.
SGI also tried a move to the Itanium, but the Itanium was very disappointing when SGI needed it to rock. So SGI had to sink far more money into improving their MIPS line in order to remain competitive. Despite this, confidence in SGI waned from these mistakes -- why blow a couple million dollars on a MIPS supercomputer when its own manufacturer doesn't have confidence in the architecture?
On the software side, the killer for them was the port of Maya to Windows, Mac and Linux. SGI bought the companies for Alias|Wavefront, and they put out Maya for all these platforms. It sold a lot of copies, but 3D companies no longer had to buy SGI workstations in order to do their work. Microsoft's only involvement was buying Softimage, which probably pressured SGI to make the purchases and port.
SGI was trying to diversify, and were even putting chips in game consoles like the PS/2, but nothing destroyed their core business like Linux did, as directly evidenced in Linux now being their primary product, but of course not leading to any financial stability.