I agree with you completely. Slavery really had little to do with the war at the onset. While Lincoln abhorred slavery he thought it should be ended by the government buying slaves and sending them home. I assume every black person today is glad it didn’t come to that.
I don’t remember where I read it, but if I remember correctly, Lincolns’ rational was that the federal arsenals and forts located in the South were the property, bought and paid for, of the Federal government. Thus he felt the Federal government had the right to do anything in its’ power to retain or regain that property. This may sound hokey, but it was the reasoning he used.
Read the ordinances of secession passed by the various southern states. A couple of them may not mention slavery as a primary cause of secession, but most do, loud and clear.