I had a personal interest in studying the origin of life when I was in college. At the Smithsonian, I plugged my (admittedly fantastically subjective) guesses into their Fermi computer. Know how many intelligent races my guesses predicted in the universe? 2.
Fermi's dilemma present evidence for their not being *hordes* of intelligent races. If there are a relatively small number, which evolved only, say, on 3rd-generation (and therefore, recent) stars, we could be outside of each other's "cone of light."
The "cone of light" refers to the area that can be travelled to within the Universe (presuming the speed of light as maximum speed). Since much of the Universe is speeding away from us at just about the speed of light, the area of the cone of light is finite, and less than the area of the Universe (OK, "volume" is the correct term here, not "area"). Since, over time, this area grows steadily larger as time progresses away from the point of origin, the area appears conical when plotted in a 4th dimension chart.
It's all very simple. The earth is like Tatooine. It exists on the outer rim and the Empire has little interest or influence in this system.