I'll allow James Madison to answer that.
I return my thanks for the copy of your late very powerful Speech in the Senate of the United S. It crushes "nullification" and must hasten the abandonment of "Secession." But this dodges the blow by confounding the claim to secede at will, with the right of seceding from intolerable oppression. The former answers itself, being a violation, without cause, of a faith solemnly pledged. The latter is another name only for revolution, about which there is no theoretic controversy.
...But this dodges the blow by confounding the claim to secede at will, with the right of seceding from intolerable oppression...
Madison is saying that it's okay if you are facing intolerable oppression. So who gets to decide if their oppression is intolerable?
Obviously King George did not think the Colonies were intolerably oppressed. For that matter, neither did the third of the population that constituted the British loyalists. Neither did the populations of Canada.
It would appear that if we follow the example of the founders, it is they who decided the oppression was intolerable, while obviously the King disagreed.
So who gets to decide? The people who want out, or the people who want to keep them in?