Until we get spectral images of several billion extra-solar planets we haven't really looked. All the rest is speculation.
Fermi's point is that we don't need to look. They would be here. Agree or disagree, but that is the argument. There are a lot fewer unknown variables in Fermi's conjecture than in the Drake equation.
Another point, made in a recent issue of Scientific American, is that only a tiny portion of the galaxy, represententing about one percent of the stars, is habitable. So this becomes the starting point for the Drake equation.