So how come the libertarian Republicans fared the worst?
I wouldn't call them that. They typically were described as moderates or centrist RINOs, like Jim Leach, author in the House of the ban on Internet gambling, IIRC.
Republican Incumbents Who Lost in the House:
Jim Ryun (KS) - ACU rating 98
J.D. Hayworth (AZ) - ACU rating 98
Richard Pombo (CA) - ACU rating 97
Chris Chocola (IN) - ACU rating 95
Gil Gutknecht (MN) - ACU rating 94
Charles Taylor (NC) - ACU rating 92
Mike Sodrel (IN) - ACU rating 92
Melissa Hart (PA) - ACU rating 91
John Hostettler (IN) - ACU rating 90
Don Sherwood (PA) - ACU rating 87
Anne Northup (KY) - ACU rating 86
Clay Shaw (NC) ACU rating 82
John Sweeney (NY) ACU rating 77
Jeb Bradley (NH) ACU rating 71
Charles Bass (NH) ACU rating 71
Curt Weldon (PA) ACU rating 70
Sue Kelly (NY) ACU rating 65
Mike Fitzpatrick (PA) ACU rating 60
Nancy Johnson (CT) ACU rating 47
Jim Leach (IA) ACU rating 43
Vacated Republican seats lost
DeLay (TX) ACU rating 95
Beuprez (COL) ACU rating 93
Green (WI) ACU rating 88
Nussle (IA) ACU rating 86
Ney (OH) ACU rating 86
Foley (FLA) ACU rating 78
Koly (AZ) ACU rating 74
Boelert (NY) ACU rating 40
Democrat Incumbent seats lost
None.
+++
Republican Incumbents Who Lost in the Senate:
Burns (MT) - ACU rating 91
Allen (VA) - ACU rating 92
Santorum (PA) - ACU rating 88
Talent (MO) - ACU rating 93
DeWine (OH) - ACU rating 80
Chafee (RI) - ACU rating 37
Democrat Incumbent seats lost
None.
Those are some rather impressive American Conservative Union ratings for libertarian Republicans.
They didn't, RINOs did, on the whole. In fact, libertarians now occupy more seats in local municipalities than we ever have. Moderate principles lose elections, that's the way things work - No one wants a candidate who can't decide where they are. And RINOs are in no way adherent to Libertarian priciples. Economically strong and socially strong conservatism wins elections.