I would suggest finding a new doctor, but if he said, "I had to ask", then it sounds like he at least doesn't agree with forcing everyone to get "vaccinated".
I got a new doctor this year after I felt like my previous doctor towed the corporate line too much.
My new doctor promotes non processed food nutrition and physical activity first. We spent more time on my sleep patterns than on vaxxes. She did offer them and I suspect it’s a requirement for the hospital system. I think she was secretly pleased that I refused.
My point is that he has changed his tune since last year. Still pushing the vax but less fanatical about it.
I’m okay w/ him as a doctor under the theory of trust but verify.