It is good of you to acknowledge that Virginia was clearly guided by the principle that states had a right to independence. It's a shame that the rest of them did not recognize this right.
So, the determining factor remains slavery. No dispute over slavery, fugitive slaves, etc. no secession.
You are begging the question here.
I'll just add this: I believe that the principal of eradicating slavery trumps the principal of self-determination. Even if South Carolina had been an independent country right from the start, the north would have had the right to invade and liberate slaves by force.
Self determination begins with the individual, not with a state or nation. Virginia, and other states that seceded, did not believe that.