Mitch wasn’t in charge in 2004. Bill Frist was head of the Republicans in the Senate and George Allen was in charge of the NRSC.
He was the whip from 2002 to 2006 (and he was fairly effective) but Frist didn’t provide any leadership. Allen made the decision to support Specter over Toomey in the primary.
No, he meant he supported Specter over Toomey in 2010.