Not doubting your specific experience, but I can find nothing on line regarding a mandatory physical if you only have Part B. Plenty about it being covered, but nothing about it being required. Excerpt from another source:
https://boomerbenefits.com/medicare-annual-wellness-visit/
The Medicare Annual Wellness Visit is not mandatory. It is a medical visit that you can take advantage of voluntarily and free of charge. The wellness visit is intended to keep you in touch with your Medicare doctor and ensure that your doctor can help you with preventive care planning.
My doc’s staff sends out the letters, so it may be just a local requirement for her medical center...............