Have at it.
What if I don't want to be affected by people who engage in anti-social behaviors whatsoever?
Avoid yourself at all costs.;^]
You see, as hard as you try, I'm still going to have to pick up the costs for anti-social behaviors in some way.
Not unless you subscribe to the same old socialist crap running rampant in the United States today. It is the argument over the distribution of socialist goodies that drives most of the dissent.
As long as I'm sharing a society with perverts, they're going to affect me in some way.
And as long as they have to share it with you, you'll effect them in some way. And neither of you consider the other's influence beneficial. But so long as neither of you violate the rights of the other by initiating force or fraud, you are both otherwise free to act as you please, and ignore the other to your heart's content.
Don't I have a 'right' to live the way I wish, around the people I wish to live around? Isn't this the essence of libertarianism?
You have the right to choose your own associations. You have the right to purchase property with like-minded others. You have the right to include or exclude anyone you wish.
What you do NOT have the right to do, is mandate or prohibit the otherwise peaceful behavior of your unwilling neighbors by force.
Again, I can't make it any clearer than that.
That would be the 10th amendment.