There is no doubt in my mind that if this was really enough to justify throwing out a sitting POTUS then Obama would have definitely been thrown out.
With Boehner and Ryan as House Speakers he wouldn’t have.