The people of PA apparently got sick enough of him to toss him out on his ass in 2006.
In case you didn't notice, that was almost entirely the fault of Bush and Rove. The Democrats swept the country in 2006, and built their congressional majority that year. Pennsylvania is a swing state, and not an easy victory for a conservative Senator, especially with a strong opponent who was a very good liar and also had the sympathy vote. Bush won big in 2004, but over the next two years he disappointed his base and threw it all away. 2006 was a disaster over most of the country.
Rick did endorse Specter, for several reasons: Because Bush and Rove, not yet discredited, said they needed him to put through more pro-life SCOTUS nominees like Clarence Thomas, who never would have made it without Specter's powerful support. And indeed Specter did push Roberts and Alito through, before he went south again. Also, because Specter had endorsed Santorum earlier, so he owed him. Although Specter then betrayed Rick and turned his back on him.
Pennsylvania happens to be a swing state, and without Specter's support, plus the base voters who were foolishly angry about the Specter endorsement earlier, plus Bush's growing unpopularity, Rick was bound to lose.