The people are sovereign. There is no higher law than the will of the people. If a state votes to leave the union, then the union can only prevent that from happening by resorting to armed force against the people.
The Constitution was established by “the people of the United States,” not by the states.
If the “people of the United States” ever decide to dissolve the Union, they have every right to do so. But the people of an individual State do not have that right, as the southerner and slaveowner Andrew Jackson pointed out with great forcefulness to the people of South Carolina.
To carry your principle to its logical conclusion, the “people of a county” have a right to secede from the state if they vote to do so. And the people of a town have a right to secede from the county. Heck, the people of a household have a right to vote to secede from nation, state, county and town.
Unless you believe a State for some reason has a sacred status that make it indivisible, while all other governmental jurisdictions are mutable at the will of the inhabitants.