The Constitution was established by “the people of the United States,” not by the states.
If the “people of the United States” ever decide to dissolve the Union, they have every right to do so. But the people of an individual State do not have that right, as the southerner and slaveowner Andrew Jackson pointed out with great forcefulness to the people of South Carolina.
To carry your principle to its logical conclusion, the “people of a county” have a right to secede from the state if they vote to do so. And the people of a town have a right to secede from the county. Heck, the people of a household have a right to vote to secede from nation, state, county and town.
Unless you believe a State for some reason has a sacred status that make it indivisible, while all other governmental jurisdictions are mutable at the will of the inhabitants.
Neither counties nor towns have special recognition within the constitution. But the states have powers that the federal government does not. I would argue that if it is the will of the people of a state that they leave the union, then the constitution does not expressly forbid this action, and therefore it is a lawful action.
Of course, tyrants believe that might makes right, so in 1861 there was a war and the legal question was eventually settled in favor of the victor. But I still say the CSA was on solid legal ground.