There is no question that only Congress can do anything with the federal courts.
In all candor, the constitution is completely dead today in 2017. There is not a single branch of government that actually observes it. And no political element in DC feels constrained by it, much less feels obligated to cherish it.
It is only occasionally brought out for use as a weapon or a pretext. Basically as an alinskite tool they use to try to force conservatives to live up to.
We have reached a point that defeating them matters more. Sad to say, but it’s reality.
Look at taxes. Imminent domain. Mass domestic spying by our military. For fun show me where a CIA is authorized. Or a the modern abuse of the Commerce clause. Or a national dept of education.
2nd amendment restrictions. Search and seizure rules from DHS and a dozen other agencies. The federal reserve is blatantly unconstitutional. The list goes on.
You are correct in your point. But also nothing on the constitution allows a fed judge to rule on laws. That was simply a power grab by the court in Marbury vs Madison.