we do not know if he knew she was married.
Well, since she was one of his employees, I assume he knew her marital status.. However, even if he didn't know, it is still adultery. It may mitigate his culpability on that score, but he was still involved in a sexual relationship with another woman while he was still married. That constitutes an affair.