You know what they say about assumptions.
And sometimes what they say about assumptions is to attack the messenger to detract from the message.
We see it all the time as a favorite MO of the Democratic Party, CAIR, ACLU, etc, etc.
You made the argument that a man might be essentially driven into the arms of another woman by a distasteful wife.
The Clinton scenario fits your criteria for justifying adultery, hence the assumption holds water.