Sumter was built on a man-made island constructed from granite shipped in from New England. It doesn't belong to South Carolina and never has.
The fort sits on an underwater sand bar that is inside Charleston Harbor. Are you arguing that the harbor and its seafloor does not belong to South Carolina?