Why are you calling out libertarians? Libertarians and libertines are not necessarily the same thing.
As a libertarianesque sort of person, my stance is that they are a private group and ought to be able to set up their own rules, but they should also be legally liable for the consequences (unless they get people to sign waivers). I imagine almost all other libertarianesque people will feel the same way. Freedom of association and whatnot.
The Girl Scouts allows adult males to be girl scout leaders PROVIDED there is also an adult female leader there too. This is a tacit admission that having adult males in charge of a group of girls (some of whom may be of child-bearing age, technically, if you know what I’m saying) is a bad idea and that another person needs to be there to make sure no hanky panky happens.
The general libertarian view is that the government should not coerce anybody into associating with somebody else. Employers shouldn’t be forced to hire a particular person, employees shouldn’t be forced to work for a particular person or organization, etc. Now, many libertarians may believe that homosexuality is perfectly acceptable and that the Boy Scouts are a bunch of backwards jerks for not accepting gay scout leaders, but they will support their right to exclude whoever they wish.
Libertarians support homosexualizing the military and adoption.