Moving the goalposts. The only legitimate authority for Lincoln to invade the South is the claim that secession contravenes constitutional law. (It does not.) There was no legal authority under the US constitution for Lincoln to free slaves, and in fact the constitution absolutely forbade it. (Article IV, section 2.)
Lincoln's only authority for prosecuting the war was based on suppressing "rebellion" which is a huge twisting of the meaning of the word "rebellion" to force it into a definition that justified unleashing his military powers on the Southern states. (Chief Justice Salmon P. Chase clarified that "secession is not rebellion." )
By the time the war had been going on for a year and a half, people had become so accustomed to doing whatever Lincoln told them to do, that he felt sufficiently powerful to take the step of singlehandedly declaring slavery to be illegal, and in direct contravention of constitutional law. Again, he used the assertion of "rebellion" to get people to ignore his lawbreaking on this matter.
You forget that little matter of violent armed insurrection.
Wrong. Murderers and slavers can be stopped. (However, you have to be pragmatic.) The Constitution only protected the south while the south was in the union. After the south seceded, God help them.