Islam is an ideology, though I do think that the ideology lends itself towards fanatical violence and systematic oppression, I think a lot of people make the mistake of believing that an individual Muslim is out to hurt them.
But again, you’re talking a set of beliefs, culture and history compared to what sex people are born.
Except for the complicating detail that is was INDIVIDUALS who flew the planes into the Twin Towers, and it's individuals who strap suicide vests to their bodies, and it's individuals who storm schools and rape and kill the children in them. The ideology can't set off bombs or break into buildings, or pull triggers.
So, yes, it makes perfect sense to worry that any individual muslim is out to hurt me.