Which is the exact attitude that killed a lot of neighborhoods in the 50s and 60s. White flight.
The first blacks to move into white areas are typically folks with decent jobs who can get a mortgage; the businessman, shopkeeper, professional, etc. But some whites flee rather than waiting a while to see what kind of neighbors the newcomers will be. When too many do so, investors buy at bargain prices and rent the homes out. Renters are less concerned with keeping up the neighborhood than are owners, so things deteriorate.
Problems also arrive when the neighbor's kids become teens, and seek to hang out with black friends who are NOT necessarily from middle-class culture.