Actually, from the self-professed swingers I've met, I would say that the situation is quite the opposite. On average they seem to have far more self-confidence than the general population. I don't know if it is true in the general case, but it is what I've observed. Those people are hardly lacking in the self-esteem department, perhaps because they've obviously gotten past any self-image insecurities they may have had to become involved in the swinger scene to begin with.