Yeah, the only Court required to exist by the Constitution is the USSC.
All others District or Appeals are creations of Congressional legislation and as such can be simply legislated into oblivion.
Good luck with that. We can't even get Congress to get rid of Obamacare.