You forget that little matter of violent armed insurrection.
But it wasn’t just Lincoln’s claim. It was also what President Buchanan’s attorney general Jeremiah Black stated that states had no right to secede and Buchanan agreed with him. On top of that Congress supported President Lincoln during the war, so those congressman did not believe that a state had a right to secede.
Also, "Insurrection" refers to a minority of the population within a nation state, it does not apply to the majority within a nation state.
Our system was a collection of nation states, each of which had authority independent of the others before they formed the confederacy under the articles of confederation.
Both New York and Virginia, in their ratification documents, reserved the right to re-assume their sovereign powers, and this ratification statement was accepted without comment by the government at the time.