Actually, it is when it can be proven IOW the rules of evidence in the UCMJ. When it is prosecuted, it's usually to the overwhelming approval of the personnel in the command!
During my career, I saw 1 1Lt; 3 captains; 1 Lt Colonel; 1 Colonel; and a Lieutenant General go down in flames for it!
I've personally prosecuted more than that for adultery. Granted, almost always the affair involves an officer and his/her subordinate, which is the underlying predicate for the prosecution. But, adultery is always one of the specifications charged.