Since his memos are of the meeting/spoken words with the President, would they not be, automatically, considered classified, until someone says otherwise?
No. They were his impressions. Like diary entries