I wear an infidel cap also on occasion. People seem a bit more distant once they notice it. Either they don’t know what an infidel is (which they themselves are also) or they see the Arabic script and think I’m some sort of jihadist.
I get the SAME reaction to it! Amazing that people are so uninformed or ignorant.
Don’t rule out the possibility that they don’t want to get hit by a stray bullet that was meant for you :<)