In the military this was termed Dereliction of Duty. And yes, the cabinet can remove Joe, but if Joe doesn't go along it will take 2/3 votes in both houses of Congress to make him.
A Cabinet made up of Dem apparatchiks is not going to do anything against a sitting Dem president. We can expect no action via that avenue.