There was no right to unilaterally destroy the Union. Not even for the protection of slavery.
In your view states could leave for any reason which illustrates the absurdity of the claim.
It would not have destroyed the union. It simply would have made it smaller. If Wisconsin wished to maintain a relationship with Vermont, nobody ever proposed to interfere with that.
There’s nothing absurd about the principle that government derives its legitimacy from the consent of the governed. That was the very principle the 13 colonies asserted when they seceded from the British Empire.
11 states left the union. The USA was still there and there was no threat of it being "destroyed". In fact the USA thrived during the war years and had the resources to complete the inter continental RR.
Think of secession like cell division and not a destructive thing.