I’m pretty sure that secession was the irreconcialiable difference. The question that a state could nullify Federal laws, and that they could secede is not a question that can be tolerated by a Federal government intent on imposing it’s central authority.
Say what you want about slavery, the Emancipation Proclamation only freed slaves that were not in Lincoln’s jurisdiction. That did not change until the 13th amendment to the constitution, passed after Lincoln’s assassination, in December of 1865. They weren’t given voting rights until 1868, the following election year.
And, the subject of secession was not resolved... only in the sense of the “winner” of the war by act of force. Legally, it is still there for us.