Years ago, I was in a church where one of the deacons left his wife and shacked up with a young woman in the choir.
It was obviously wrong.
However, in private, our pastor made the comment, "If you had ever seen how his wife treated him at home, you wouldn't be so quick to condemn." Found out later that his wife was one of those who ALWAYS found fault with her husband, and vocalized it constantly. He didn't leave her for sex, he left her for someone that actually LIKED him.
It was still wrong, but it taught me to be careful about condemning someone based on the obvious facts.
I agree with you and have said that many times on FR. In fact, in many cases a spouse that cheats has been alienated from their spouse and seeks what they either can't or won't give them. It's hard to blame someone for that.
There mere fact that a spouse has been cheated upon doesn't make them a saint. Anybody with a mature world view, and capable of interpreting facts just a little more complex than the average sunday funny pages, should be able to know that.
Yet, far too many people can't cope with it. A cheating spouse does wrong, no question. But that doesn't mean the cheated upon spouse is completely in the right, either.