It's selective and self reflexive, I think. The stigma is there, but it's not uniform. A lot of women, for example, would rip into a cheating husband of their friend, or a cheating male friend of their husband.
At the same time, they might be more sympathetic to one of their female friends who is banging a hunky guy on the side.
On an even kookier quirk, a single woman might find the type of man who cheats as agressive and sexy.
A man might rip into a cheating wife of one of his buddies, but not be nearly as judgmental to one of his male friends having a fling with a sexy girl.
It's all based on pride, and selfserving rather than being self-sacrificial and giving themselves to their wife/husband.
You, sir, are hanging around the wrong men.
I had (note the past tense) a friend who cheated on his wife. The urge to leap across the table and throttle him as he (tried to) justify cheating on his wife was nearly overwhelming. He proved himself untrustworthy in his marriage, and (what a coincidence) went on to commit fraud in his business and (other) personal dealings, too. He lost his marriage, his business, and his friends, because he believed his cheating ways were all justified. He's defaulted on loans, forged signatures, and occasionally "appropriated" unattended money. He's fathered an illegitimate child. He pawned his wedding ring (how symbolic is that?). And he has told me that he doesn't need to repent of anything. No one will loan money to my friend to bail him out of his troubles (and those that have were never repaid). Everyone has seen, first-hand, that he's untrustworthy, in every meaning of that word. He has no friends, except those who see him as a charity case.
Conversely, I know a man who talked a married friend out of "having a fling" with a willing old flame. The friend now credits this man with saving his marriage, and the two have maintained strong friendships and marriages in the years that followed (and their credit scores are sky-high). Bottom line is that they trust each other, and they are trustworthy.
There's a reason married men (ought to) wear wedding rings. They serve as reminders of a solemn vow we took facing a beautiful woman, that we would always honor that woman by being faithful to her. Men who break their vows are more than adulterers - they are liars and thieves as well.
"It's selective and self reflexive, I think."
I think you're right.