Our pediatrician takes about 1.5 seconds to look "under the underwear" of my daughters, and this only after asking them if it's okay to look. I don't think it's about checking for abuse as much as ascertaining the onset/level of pubescence. And I'm right there beside them, so I have no qualms. If something beyond this is going on, you should ask the pediatricians about the purpose of the exam. It can always be refused.
You say you don’t THINK he’s doing it because......I believe I’d ask him why he’s doing it. I would want to know exactly why.