I, too, am a counselor. I accept no female clients for just this reason. In my previous career, as a manager in a technology company, I saw another guy fasely accused of sexual harrassment. After that, although I couldn't refuse to meet with female employees, I had a large window placed in the door to my office.
Smart move.