This is like saying that if my employee lied to me on his or her employment application I have no recourse but to continue to employ them is it not?
You are assuming that voters have final say in who their elected officials are. They do not.
The framers enacted a system where originally the only federal official you directly elected was a Representative. And if you need to fire them, you get to do it in two years.
Senators used to be voted on by state legislators, not voters.
And with Presidents we have the Electoral College which has been bastardized.
The Electoral College could have demanded proof of eligibility. Same with Congress when certifying the vote. Congress can still impeach him.
You are left without a direct recourse, but not without a recourse.
Again, think about the path you are advocating.
Want to stop the Iraq War, Code Pink? Sue George W. Bush in federal court.
Want to stop tax cuts? Sue Ronald Reagan in federal court.
It would be endless.