There are. But the Libertarian party seems to attract more than its fair share of them, particularly when you think of its size.
This is easily verifiable by simply reading what passes for "libertarian discourse" these days. Try reading that "Premier Libertarian Forum", LibertyForum for a while and then tell me in exactly what respect it differs from, say, Stormfront.