In the article, the author postulates that if aliens exist we should be overrun by probes by now. The rebuttal to that is that either they have different methods of probing than we have come up with (hey, they're aliens, they think differently), or, like ours would be, theirs are small enough that we have never noticed one (it might not have come close enough to us to be recognized).
Just thought I'd throw that in. I tend to think it is a combination of distance and incomprehensibility that has kept us from detecting/recognizing our neighbors.
Of course another possibility is that a "colonization" took the form of genetic seeding of planets. In that case the answer to Fermi's question would be "You're looking at them".