Whether it's marginalized or not, the Libertarian Party has a major problem: it cannot remain true to its ideology in cases where there is not a general agreement among the people -- not only about the roles or limits of government, but also about morality.
Beyond that, you're right. There is indeed a strong tendency for libertarians to self-marginalize by defending behaviors that are harmful to the greater community (which libertarians often claim do not exist).
Excellent point.
So, what are YOU doing about it?