The actual medical file is the doctors property. You can get a copy of it but if you try to walk out they’ll report it stolen. They have to as it would be reportable to HHS.
It's been many years, but I got my entire ORIGINAL primary care folder at one time.
I would still advocate firing your Doc, rather than answering such intrusive and irrelevant questions.