You do when you’ve committed to an agreement with them.
I can’t buy that argument because it simply eviscerates the right. Everybody is in some sort of agreement with their current government, whether explicit or implicit. So what you are arguing effectively puts us in bondage to our current form of government, since governments rarely agree willingly to the grievances of citizens who wish to separate.
If you give the state a veto over this right, it will almost universally exercise it, and then you have no right at all.