I may disappoint you, but I don't call myself a 'libertarian' - I happen to favor, among other things, universal military service, significant import duties and strict immigration control. But, seriously, do you have any example of 'nutty' libertarian ideas? I disagree with some of theirs but I find debating them to be intellectually stimulating. Calling someone else's views 'nuts' has no value and does not bring us one inch closer to the ultimate truth. However, it does mark those who use such labels as unworthy of being respected or even taken seriously.
I rarely heard people stating that the libertarians were 'nuts' because [...]. I do know that most who state that the libs are 'nuts' have no idea what the libertarians stand for but were told that they were nuts and, absent any first-hand information, simply repeated what they were told - which would place them into the ever-popular and always in-the-majority 'sheeple' crowd.