That may be the case. Remember, these black holes are millions of light years away. This means what is being observed is the state of things millions of years ago. We cannot know what is going on now; it’s quite possible that over those millions of years, these black holes have cleared most matter from the surrounding volume of space
Do we even know what a black hole is?