I've asked that point many times in many debates on this topic. I keep asking the Union defenders "why would you want to keep a bunch of slave states in your Union? Why wouldn't you say "good riddance!"?"
What I learned several years ago is the slavery argument is just misdirection. The real truth is that southern secession would have caused a massive economic loss to the rich and powerful men in the Northeast who had strong influence on the Government.
Over the years, New York and it's environs had acquired almost total control of Southern export products to Europe, and they had protectionist laws put in place that required all the Southerners (Northerners too) to buy their products in the place of cheaper and better quality European manufactured goods.
With the South leaving, it would have cost these northern "robber barons" about 750 million dollars per year in losses, and that is in a 4 billion dollar economy.
Southern secession was an economic disaster for the corrupt influencers in the Northeast. This is the real reason why the North invaded the South.
They didn't care about slavery, and we know this because they voted to approve the Corwin amendment which would have preserved slavery indefinitely.
What they cared about was money, and *THAT* is why they invaded.
Here is a map that illustrates my point. 72% of that money represented by those trade tariffs came from the Southern states.
Again, the vast majority of that money came from the South. With Southern secession, all that trade would have moved from New York and Boston to Southern port cities.