Eligibility of the President is a political question in that a majority of Electoral College voters can elect anyone, even a person that is ineligible. At the state level, voters are voting for the electors and not the specific candidates. You are right in that Congress can only affirm or deny the counted votes and not the eligibility of the candidates.
SCOTUS has repeatedly opined the natural born citizenship requirement is undefined and not specific, affirming eligibility to be a political question with respect to an election and swearing in.
Eligibility becomes a matter for adjudication when a person claims a usurper has caused them specific and a particular harm when the usurper sign a law or executive order which impacted the complainant directly, i.e. a plaintiff was sent notice he/she must pay a fine for not obtaining health insurance after the ACA was signed by Obama. The plaintiff will have standing to object to a usurper directly and negatively impacting their life with a law signed by the usurper. After standing is established, the usurper will have to prove eligibility or the plaintiff will absolved of paying any fines or penalties.
A plaintiff with standing cannot have a usurper removed from office because a majority have voted to put the usurper in office. Each person, individually, must object to the usurper after they have suffered a particular harm or threat of harm due to the actions of the usurper.
Waivers run the clock out for the usurper until he leaves office. No one person with a waiver of fines or penalties has standing to sue the usurper. Once the usurper leaves office, the DeFacto Officer Doctrine indemnifies the U.S.Government. The laws of the usurper cannot be challenged by the fact a person objects to a usurper imposing fines or penalties on them after the usurper leaves office.
When has SCOTUS said that NBC is a political question?