Sumter wasn’t “annexed” - it was illegally seized as an act of direct provocation. An incredibly stupid thing to do. They that did it have no room to complain about the union forcefully taking it back.
From the moment the South got the idea to go to war they were doomed to lose.