Where are you getting your information? It seems to me that especially since the Badnarik mistake, more libertarians are leaving the LP for the RLC than the other way around. Note that "leaving" would be interpreted loosely, regardless of the direction of the shift. Inherent in being a libertarian is an individualism that eschews blind loyalty to a "group". Thus individual libertarians are as a matter of convenience alligned with a party for a period of time.
The LP has at least 3 challenges.
1. It (its remaining members) compromise on some ideological points to build a coalition but not on other ideological points that would also enhance the coalition into a more viable/larger group. This makes the debate over purity vs compromise absurd.
2. It lacks leaders of the quality of Hosper, Paul, Clark. Harry Browne and then Badnarik were not leaders who would attract a broad base. This proves that "leadership" as exhibited by non-ideological types like Guiliani, Rumsfeld, etc does make a different, despite what us ideological types would like to pretend otherwise.
3. Any 3d party must build on the negative feelings toward the major parties. Despite all the non-libertarian and non-conservative activity of the GOP President and GOP Congress, the LP has had no vision to attract those of us unhappy with the "spend and borrow" policy of Frist-Hastert. Likewise with other opportunities blown.
The LP has also failed to exploit the mistakes of the Democrat party. The Moveon/Deaniac crazies have taken over the Democrat party and left many sane Democrats without a party. There is no way the LP could outflank the Democrat crazies and be more wacko. But that is what some LP types have tried to do. Witness the absurd cries of "Bush lied". To this day, I have yet to hear a specific quote in quotation marks of a Bush statement on terrorism or Iraq that is a lie. (Bush did lie about Katrina.) Yet some LP types continue to try to outdo the Democrats in going wacko.