I think Fermi might well be a smart man. Figures do not lie, but liars always figure. Anyone can cast a slant on the facts, remember? “if the gluv done fit you gotta acquit?”
I also think if we start with the premise that we are still cavemen with caveman tools we are probably closer to the truth. There are far greater paradoxes of space and time travel but the chapters are not organized that way. Try reading the Bible.
To think of all the planets that exist and then suggest that we are the only one that has life is a little narrow minded and short sighted.
Actually the converse of your statement is true. It is hard to accept but we are alone in the universe. All these Earth like planets are just pretty marbles.....