It was probably his association with Roy Cohn. Smearing him because of his associations.
The drag stuff is ridiculous. And, yes, it’s amazing what people believe.
I believe you are correct about Roy Cohn being the basis for that.
Someone puts out the lie, and others repeat it, and so it goes.