That's a silly claim! Fermi's paradox makes NO SUCH assumption.
If someone tells you that the earth is inhabited by dragons and you ask where they are, you haven't made an assumption that they do not exist. You're simply acknowledging that if dragons inhabit the earth, we ought to see evidence for them, and that the burden of proof is in on the guy who says otherwise.
The claim is A) not mine, but B) not silly. The burden of proof is on those who claim that anything they haven't experienced never happened. That's what "skeptics" about UFO sightings have in common with those who deny the Moon landings ever took place.