You'll not find "sworn into office" in the Constitution. Yes the Constitution requires that the oath of office, which is specified, word by word therein, be taken "Before he enter on the Execution of his Office", but it does not say the he becomes President upon taking the oath. What it does say is that having satisfied all the requirements. Win majority of the electoral vote, be 35 or over, be 14 years a resident, and be a natural born citizen, then he becomes President at noon on January 20th following the election. (since the XXth amendment of 1933)
The inauguration is not a coronation. It's only traditional, not Constitutional, not being mentioned anywhere in that document.
Congress has the power to remove a President, but logically someone not eligible to the office cannot be President.