You fail to take SamAdams main point. In the sense that the planets most close to Earth are simply to far away in terms of light years in that we don't know where they are, have never seen them, and never will.
If intelligent life is common to the planets close to earth, then at least some life there should have had technology for at least a few million years which is less than a second in geological time and a few million years should be more than enough time for them to find us. Granted, we have been able to reach out and find them for only a few years, but if it is common, someone should have found us. We are rare, very rare, maybe divinely rare. Further, I was an atheist, until I studied physics in college. Then I realized how unique we really are.