For a State to declare adultery a crime is for the State to promote the idea that a contract between a man and a woman and recognized by the State is binding and moral. For someone to break that contract and the State, which recognized it's existance, to turn a blind eye is to encourage a view that the State does not hold it's responsibilities to contractual endorsement very seriously.
Can you see where this can lead? If a man's wife has an affair and the State does nothing to try to penalize her for her breach of contract, how then can it penalize him for any breach of contract that he might persue in regards to his business dealing? Sure his business partner might be able to sue him, just as the man might be able to sue his wife. But can the State, in good conscience, punish him for breaking a contract that the State recognizes when the State is not willing to punish his wife for breaking a contract that the State recognizes?
Do I think adultery laws should be aggresively enforced? No, not really. Do I see the benefit of having them? Yes.
And that's a reason that gives legitimacy to a law against adultery where a law against sodomy (assuming unmarried consenting adults) is not.
No one's rights are being violated in the sodomy relationship. Certainly there is no specific right to engage in sodomy in the Constitution, but I believe it certainly falls under the ninth amendment.
The enumeration in the Constitution, of certain rights, shall not be construed to deny or disparage others retained by the people.
There's no Consitutional right to pick your nose. Should it be criminalized?