So carrying your reasoning a little further: States with equal authority cannot be judges of their own case and any state can leave if the contract has been violated, given that the leaving state does not have equal authority and therefore can be a judge of its own contract violation case in which circumstance the other states may also be judges by virtue of the aforementioned lack of equal authority.
I did from posts 900 to 1150. That was a mere synopsis. The fundamental problem is that if the contract has been violated, those who violated it cannot be judges of the case. There is no higher authority (unless --I dare not think, less utter it -- the UN). Since each state is sovereign (wherefore we each live under 2 constitutions), each state which in principle is prior to the union of states has proper authority to secede. It is not right (nor anywhere prohibited by the US Constitution) for the one aggrieved (or allegedly aggrieved) state to be at the mercy of the those that broke covenant. The sovereign parties who entered such a covenant or contract may leave if the terms have been violated. Who is to judge? Each party, unless there is a higher authority to whom to appeal. However, we have none such unless you think we should go to the Pope or to the UN. It's a simple principle of justice.
Why, then, is it so controversial? For the obvious reason that there are grave consequences for the break up of the Union, not to mention consequences to those with imperial ambition. However, the consequences to not breaking up are far greater. I say life, liberty, and property before security.