He's also assuming that we've never encountered aliens. Much as I hate Van Daaniken if you take his rather bizaar view of the world we clearly have. If Van Daaniken is right (God I hope not, not that the implication bothers me, I just don't like the idea of that pinhead being right about the time of day, much less having his "theories" proven) all of our "supernatural beings" through out recorded mythology and history were actually aliens.
There's also an assumption that were aliens to stop by they'd decide we were worth dealing with at all. Again cosmosological centrism shows us through China's history that's it's entirely possible to go someplace, even a great distance that would be difficult to travel, and decide it was a complete waste of time, go home and never come back.
Finally, he's assuming this empire would develop unopposed, or even want to expand that wide in the first place. Looking at how we've performed in 20th century wars (Patton's 3rd Army being a wonderful example) you could draw a very logical conclusion that a civilization as advanced as our could easily conquer an earth sized planet in just a few decades. The fact that America is not a globe spanning nation with total ownership of the planet does not disprove that America exists; it means that we were opposed and really not into that whole globe spanning empire thing.
This is all an example of assigning experts too much knowledge. No one challenges that Fermi was a brilliant man, within his field. But this assertion of his shows that clearly he lacked understanding of sociology and psychology (much like Einstein with his line, very popular amung liberals, that you cannot prepare for war and peace at the same time; clearly demonstrating his lack of understanding on how to achieve peace by scaring those who would make war against you), not to mention clear misses in understanding how technology advances and the rates at which it happens.
The problem with this view is that it almost assumes that technological improvement is a linear or sublinear curve, when in fact it is exponential.
This is something to think about very careful. Let's assume an absurdly narrow window of all the civilizations in our galaxy starting within 100,000 years of each other. Let's also make the absurd assumption that technology in all these civilizations grows at exactly the same pace. Even with these assumptions, the truth of the matter is that a civilization technologically only 200 years ahead of us will effectively have god-like powers from our perspective today. The difference between now and 10,000 years of technological advancement is so great that it isn't even imaginable. So even if we were the average civilization in the local neighborhood, we should see the galaxy swarming with species of unimaginable power, the guys on the right-hand side of the technological bell curve.
But we don't see that. This leads to a few possible conclusions, most of which you'll probably find distasteful. One, we could be the most advanced civilization in the universe. Someone has to be it, but it seems improbable because a lot of civilizations could've had an earlier start by at least a few billion years. Second, there are no other civilizations. Third, all the other civilizations have died. Fourth, the only other existing civilizations haven't generated evidence within our lightcone. I can offer a few more, some of which are far more probable than some of the ones I listed here, but they open up their own can of worms.
There is a variant of the Fermi notion. It does not require an interstellar "empire", no conquering:
Given an advanced technological civilization, say one 10,000 years more advanced than we are, one strategy which will suggest itself is the construction of "Von Neumann robots," which are sent at random to the stars. Say at 5% of light speed. These robots carry the instruction: "Learn all you can about the solar system you enter. Using local materials, build a copy of yourself and download all of your data into it. Then both robots depart for different randomly-selected stars."
This results in an exponential explosion of probes. The pay-off comes when one probe--a descendant of the first ones--wanders home at random and dumps its datastore--consisting of ALL of the data collected by each ancestor.
This is a high-payoff strategy for a very small investment. If it can occur to us it will occur to at least a few of the intelligent ETs. If we are around 10,000 years from now I am certain we will attempt it.
There has been ample time--even at .05 c--for every single star in the galaxy to have been visited at least once. If high-technology civilizations are common, there should be a veritable traffic jam of probes.
We do not observe such a traffic jam; hence there are probably very, very few intelligent and technological civilizations...a conclusion corroborated by Rare Earth-like arguments.
--Boris