I assume you’re talking about blacks encountering Jews who identify as Jews by wearing a kippah or tallits, etc.? Yes, that may be true. I’m thinking of everyday life in what you refer to as a safe zone - most of NYC. As a woman, I don’t like being in bad neighborhoods or the trains late at night myself - we’re all vulnerable.
Basically, yes. As a man earning a living and living outside of the safe zones, I had no choice but to commute late at night and walk through black neighborhoods with a kipa. The whole of NYC is far from a safe zone. Upper East Side and 13th Avenue in Borough Park, or the like, sure. But those de facto sheltered safe zones are narrow, and the contrast is stark.