My previous pastor did the same thing. Either the door was open or, more frequently, a female from the congregation was in the room if he was counseling a lone female.
When our new Pastor’s office was built, the part housing his desk was made rather small. The outer office, however, has a table for four, a small couch, and a bookshelf. It also has a door that can be fully closed - in a GLASS WALL.