More sex partners in marriage certainly could cause the spouse to be less happy.
No sex partners during marriage makes for an unhappy marriage.
Or at least very tired.