Pretty sure he can’t be removed by Congress as he serves at the pleasure of the President and the Senate approved him.
Only way he gets out is resign or prison. And who’s going to prosecute him?
He can be removed via impeachment (but removal requires two thirds of the Senate).
William Belknap, who was Secretary of War under Ulysses S. Grant, is the only cabinet officer to have been impeached. He resigned upon being impeached. The Senate went ahead with a trial anyway, but Belknap was acquitted (removal would have meant disqualification for future federal office). There had been debate in the Senate over whether it had jurisdiction, given his resignation. It is thought some of the senators voting for acquittal did so because they believed they lacked jurisdiction, not because they didn't think he deserved removal.