Frankly speaking the vast, vast, vast majority of white people have far more better things to do than to hassle black people.
And in most of the country, it is very easy for us to avoid them. In northern New Jersey they tend to concentrate in a handful of towns that you have no reason to visit.