Why should they withdraw? It was their fort.
If the North had chosen to let the South be, would there have been a war regardless?
If the South chose to start one. Which they did.
This sounds like “you’re in the Union d_mnit, and you’re not allowed to leave”. If you want to make that argument, make it. You sound like you know your history better than I do, so maybe it’s legitimate and the Constitution was a one way contract, and I just don’t know better. Sumter’s in Charleston harbor, so I could see why South Carolinians might have a problem with a fort operated by what became a foreign power sitting in the middle of their shipping lanes. Sounds like it was an artificial construct anyway. Maybe the North should have volunteered to demolish it and left.