You have the right to join with others who believe as you do, buy common property, and prohibit homosexuals to access that property, or to engage in commerce with your like-minded compatriots.
What you do NOT have the right to do, is restrict their otherwise peaceful behavior in THEIR homes, businesses, and associations, by force.
I can't make it any clearer than that.
Let's say I want to form my own society with other social conservatives. Let's call it Bluenoseprudeland. And then you libertarians and all your drug-addled, pervert friends can form your own society, Transvestitistan. That way, we can both be happy and live around people we like.
Don't I have a 'right' to live the way I wish, around the people I wish to live around? Isn't this the essence of libertarianism?