That all depends on many factors. Too generalized. But based on what passes for all-black neighborhoods these days, yes a white man should fear being there. When I was in I.T. I managed a crew of techs, some were black. One lived in the ghetto, and never wanted me to visit him there. He was a tough guy, owned a shotgun and he himself was afraid of the all-black neighborhood. I appreciated that he was looking out for me and I looked out for him. Upper managers didn't like him because he looked ghetto and talked the part, but the guy really knew his networking and was smart. I helped him get into a systems engineering job. No one should do harm to someone based on race, unless they're first doing harm to you based on your race, that's my motto.
Part of that scenario happened to me in Buffalo, NY, on my way to Canada. I think I was lucky to escape.