"If this guy is right, though, then clearly they could get here, so where are they?"
If Felder is right is not indicative that they can get here. As a minimum, they must know of Felder's equation. Like you said in your post, life may be out there but they don't know how to get here.
Well, but if we can figure it out, then the assumption is that there are others out there, more advanced than us, who could also figure it out.
Obviously one or more of the assumptions are wrong. Personally, I think it's the assumption that you can move interstellar distances. People just do not comprehend the distance we're talking about.
Although the counter to that is that even if you can't travel those distances, you still should be able to send a radio signal over those distances, and so far, we haven't detected one.