I had a black guy tell me the same thing. People like to be safe.
If I, as a white man, said that I liked to live in white neighborhoods it would be construed as a racist statement! Racism is a one way street. NOT!!!