The Hartford Convention voted against secession, and the other crises you mention didn't get very far either. There were other secession scares in 1820 and 1850, which were averted by the compromises of those years. But such crises were a bit like childhood diseases. They produced antibodies that made the county stronger.
Confronted for the first time with the idea of secession many Americans rejected it as impractical, impossible or unconstitutional. That doesn't necessarily mean that they regarded the union as unbreakable, but they did reject unilateral secession by states.
There's more here in an article on how the secession crisis of 1850 produced greater devotion to the Union among many Southerners. That committment didn't last. Ten years later, secessionist arguments prevailed in the South, but the article does indicate how much gets left out of neo-confederate readings of history.
It's simply not true that before Lincoln came along the union was generally regarded as a "voluntary union of sovereign states" from which any state could leave any time it wished. When the issue came up, there was much disagreement about secession, as Jackson's and Madison's comments on the South Carolina crisis of the 1830s indicate.
"The Hartford Convention voted against secession, and the other crises you mention didn't get very far either...."
Two of the secession movements initiated by the New England Federalists were probably largely motivated by an almost pathological loathing that the Federalists harbored for Jefferson (similar to the way I felt about Bill Clinton). That may account for why they utimately did not succeed. It nevertheless reamins a fact that in all three cases of Federalist initiated secession movements the entire scope of the debate centered around questions pertaining the viability and prospects of the soon to be disencumbered states. At no time did these debates take up the issue of the right of a State to secede. That this right was inalienable and belonged to the sovereign state was unquestioned.
That this was case should not be surprising as the United States had been founded on a secession of a smaller political unit from the British Empire. To repudiate the right of secession would be equivalent of repudiating moral founding principal of our nation.
"Confronted for the first time with the idea of secession many Americans rejected it as impractical, impossible or unconstitutional..."
It is true that when any thought was given to the issue of secession it was presented in the context of being an absolute last resort against an unbearble tyranny. When this issue arose practical questions regarding the vibaility of the new political unit arose at the same time as an inevitable consequence.
"It's simply not true that before Lincoln came along the union was generally regarded as a "voluntary union of sovereign states" from which any state could leave any time it wished..."
The right of secession was understood to be reserved as check against only the most serious and sustained abuses suffered by a sovereign state. The secession crisis caused by the War of 1812 was occasioned by the abrupt and near complete rupture of trade inflicted on the New England states because of the War. In this case nor in any other case was secession invoked over a trivial political matter.
The Declaration of Independence made its case for secession from the British Empire because of a "train of usurpations". This is exactly how the Confederate states felt in 1860.