Maybe their wives don't like the idea of having sex with adulters?
“You have heard that it was said, ‘Do not commit adultery.’ But I tell you that anyone who looks at a woman lustfully has already committed adultery with her in his heart. Matthew 5:27 & 28 NIV
opps
adulters = adulterers