Conservatives are routinely called "Nazis" or "fascists" when they have absolutely no sympathy for either of those political movements and actively detest them.
Leftists bridle at being called "socialists" or "Communists" even when they espouse Marxist doctrines, invoke class warfare and wear Che Guevara T-shirts.
An opponent of affirmative action is in no way a "Nazi sympathizer". But someone who supports Castro is quite explicitly a Communist sympathizer.