I guess you're willing to overlook the occupation of the island in Charleston Harbor.
You must also be willing to overlook the invasion of Mississippi and other States of the Confederacy.
The history isn't pretty.
ML/NJ ("Honest Yankee')
Why would we overlook it? The fort and the ground it stood upon belonged to the federal government.