They could actually be fairly close and you still might not see one because it’s absorbing more of the sun rays than reflecting it.
How often have asteroids passed by earth and we only knew about it afterwards?
Also true.