If his argument that it is his constitutional duty to "Preserve the Union" is correct, he could have started it to end a secession for any reason.
His argument was that secession was illegal, not that slavery was illegal.
Would the men of the North have been willing to enlist to kill Southerners just over secession without there being a great issue such as slavery?
You are aware that many of them were drafted, and did not want to fight? One of their arguments of the time was that their lives were worth $300.00 while a slave's life was worth a thousand.
They had the worst riots in our History in the north because of the draft.
For that matter, without the big issue of slavery, would the southern states have bothered to secede? Sure there were other issues, but they were small potatoes relatively.
From what I have learned of the conflict just in the last few months, those "small potatoes" issues were intimately tied to the overall picture. Apparently the US Federal government was running on money generated from Tariffs, with the vast bulk of the costs falling on the Southern States. Anywhere between 50% to 80% of the revenue raised to support the FedGov, was the result of imports to balance Southern agriculture exports.
That Southern 20% of the US Population was paying greater than 50% of the costs of running the US Government. Port Cities like New York were making a fortune trans-shipping southern cotton as middle men. Much of the economic activity of the period was tied up in Slavery one way or the other.
You should read this and other messages from Pea Ridge regarding this period of history. They were an eye opener to me.