You've just highlighted the problem. In your opinion we can't go negative on someone because it will be labeled racism, sexism, homophobia, etc. You remind me of prosecutors I worked with who were afraid to try and introduce evidence for fear the defense would object. Those prosecutors gelded themselves just as you are doing.
The tactic of the weak is always to get the strong to nullify themselves.