It would if he was drunk or stoned. Unfortunately, it would be a miracle to prove that.
He would not be removed from office for being drunk or stoned.
226 years of constitutional history shows that, unless he committed or ordered to be committed serious crimes that he will not be removed.