It's well on its way to being criminalized, in case you hadn't noticed. Even writing an e-mail letting someone know you feel that way about them is being classified as a "hate crime" in the case of, for instance, CAIR. CAIR reported "massive wave of hate crime" against Muslims after 9/11. When pressed, turns out most of the "hate crimes" were negative e-mails. While no one has been prosecuted for that yet, as far as I know, if they are calling it a hate crime and no one is arguing with them, it's only a matter of time.
Or think of the "hate crimes" of spray painting hateful messages on public property, dorm room doors, that sort of thing. If you spray paint "Kilroy was here" you are guilty of vandalism. If you spray paint "Muslims go home" you are guilty of a hate crime. Yet the only difference is in the emotion expressed.
I stand by my assertion that your assertion is inaccurate. Anyone, like CAIR, can call something a hate-crime. That doesn't make it so. And 'might be' doesn't count.