Not at all. And there is nothing to say that the North couldn't have seceded from a slaveholding country under the idea that the contract of life, liberty and happiness had been violated. But that wasn't the axe the North had to grind.
It would have been interesting to see what the reaction of the seven deep south states if the rest of the country had said, "We're leaving. We're taking whatever federal property we can get our hands of with us. You've got what property's left and the national debt. Good luck paying for it. Good bye." The southern states would have pooped a brick.