And that's a reason that gives legitimacy to a law against adultery where a law against sodomy (assuming unmarried consenting adults) is not.
No one's rights are being violated in the sodomy relationship. Certainly there is no specific right to engage in sodomy in the Constitution, but I believe it certainly falls under the ninth amendment.
The enumeration in the Constitution, of certain rights, shall not be construed to deny or disparage others retained by the people.
There's no Consitutional right to pick your nose. Should it be criminalized?
So while I would agree that a Federal anti-sodomy law would be very easy to challenge, a State law has a different level of engagement with the citizen. Likewise, a county or town ban on, let's say pornography, has more legitimacy than a state ban would have. The closer to a citizen, the more legitimacy a legislature has concerning moral codes.
This assumes of course that a higher level of government has not secured the right. For example, if a State has legalized gay marriage, a locality can not refuse the right. But if a State has made no statement on an issue, a locality has great flexability. A town may, for example, forbid more than 6 non-related persons cohabitating in a residence with only one bedroom.