We reverse this idea in what I'll call the Standard Model of Science Fiction -- the view that humans, were we able, would expand to occupy all habitable planets within reach (unless or until stopped by an alien counterforce). People will always seek new worlds to colonize, or to which to flee. (The big question is whether it is the Muslims who would stay, or go; or liberals vs. conservatives, etc. I think we would separate if we could ....)
No it absolutely DOES have to do with aliens being able to detect us. Just because they could have gotten here doesn’t mean they have a reason to. The biggest reason to is because they found out about us. Since they would have to have some part of their civilization within 100 light years of us AND they would have to use communication technology that would detect our radio waves (extremely weakened) OR have a telescope aimed AT US that could detect man made lighting. Without being able to detect us we’re just another rock, one they might or might not find any interest in. And since we have no ability to detect them we have no idea if they’re right next door and just aren’t interested in us. Think about those tribes we occasionally find in the middle of nowhere Amazon or Congo, most of them tend to be just a week or two away from “civilization” and yet they manage to avoid detection for decades. If we can manage to not find other human beings living within a few hundred miles of us, thinking that we would detect aliens in space, even ones “next door” on a galactic scale is just a bad assumption.
The only real paradox in Fermi’s Paradox is that he was so bad at math and comprehending just how big space is. Also how big time is. We’ve got no real way to know if they were around before us and left. Again look at our own experience with ourselves, we have an entire school of science dedicated to finding previous installments of our own civilization and figuring out how they lived because we lost track of them. We lost TROY of all things, and Fermi thought we’d just KNOW if aliens had been hanging around here.
Our own history on our own planet shows aliens could be living in our own solar system and we could miss it. While I don’t think that’s the case, to think that our inability to detect them means we’re alone in the universe (note, Fermi only postulated the galaxy) is frankly stupid. On the galactic scale humans are the American Indians of the 1200s, Europe could be 2 stars over and we’ve got no way knowing.