I can see an argument for executive privilege here. Presidents need to be able to assure those they ask for advice that confidentiality can be kept. Just as the content of Oval Office conversations must be private, one could argue that even the list of persons present should be private as well.
You’re absolutely correct that presidents should be able to have confidential conversations and that also means withholding the identity of the participants in a conversation.