Not really. While I agree that not all body builders use steroids, a large percentage of them do.
18.8% use steroids.
Hardly “most”
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3905495/
And, yes, it’s aggravating. I work out a lot and have heard the whispered “steroids” bs for 30 years. No, I work out hard, eat right, and am not a pussy. I suppose it should be a compliment, though.