"I would argue that this phrase from Amendment XX applies even after the swearing in of the President."
And I would argue it doesn't. It explicitly states "President elect"....what happens after that should be nothing less than the asshole being led out of the White House in handcuffs by the MP.
I think we're of the same mind here. I also don't believe that it's necessary (or even legal) to impeach a President who has been shown to be unqualified. If he's not eligible to hold the office, then he was never President to begin with, and has been operating under false color of authority the whole time.
A simple arrest would be all that's necessary, in my opinion, should it be proved that he is ineligible to hold the office.