It has to be done while he still holds the title, not post-presidency.
If he took them with him, wasn't he president until he left?
January 20. Did he stick around in the White house after that date? I don't think so.
Ergo, when he took them, he was still president.