Let us say for the sake of argument that you are correct. Given that Slavery was legal and recognized by the Union as legal at the time, how does this justify an invasion into the land of people who want to rule themselves?
The Confederate States of America committed an act of War against the United States, when they fired on Fort Sumter.
That act and Davis’s authorization of letters of Marque and Reprisal against U.S. shipping were all that the Lincoln administration needed to justify the use of armed force to put down the “rebellion”. Jeff Davis gave Abe Lincoln all the ammunition he needed to make war on the Confederate States.