Yes, it is a problem that the Courts are so far removed from popular control that the governed cannot be said to have given consent, but to replace the Rule of Law by Rule of the Executive would only make it worse. What is needed is that the people or their elected representatives be able to overrule these egregious Court decisions.
There's a hole in your argument that I can drive a truck through, and that hole was made by the use of the word "arbitrary." You act as if Colson is saying the president may reject a SCOTUS ruling if they say toe-may-toe and he says toe-mah-toe. But "arbitrary" is hardly the issue.
For example, Lincoln defied the SCOTUS because they had overridden the clear intent of Congress on a very thorny issue that had been passed after decades of debate. Moreover, the Congress had passed the Missouri Compromise in an effort to save the Union, and the SCOTUS was undermining that just to make sure dipstick slave owners didn't have to worry about losing their "property."
You seem to be missing that we're talking about a situation where the SCOTUS defies the Constitution and the clear will of the people, and does so in a way that endangers the Republic. That's hardly an "arbitrary" situation, and it is a situation where the POTUS and Congress must act to preserve good order and separation of powers.