Actually,it happens because the debater on one side of the argument continues to restate a conviction. He is not really debating and he cannot make any compromises or moderate a conviction. Nobody can! It is the nature of a conviction.
At some point, the argument or actually a lack of true debate causes ulterior motives to be tossed into the mix.
That's fine, but the civil outcome as it should be would not end up with charges of racism when is there none but rather a stalemate, or the more polite: "let's agree to disagree".
The frustration from those on the other side comes from the fact popular opinion is not with them, and besides name-calling in the past has proven effective at shutting people up.