The fortifications were built long after we won our rebellion against the Brits. Sumter was built upon shoals using granite. Until the US government constructed it it didn't exist. So no - the Brits never owned it.
Owning the land is the most important aspect about owning what is on top of the land. I'm pretty sure the British Owned the land. A trivial point anyway.
The Question remains, do people have a right to to "dissolve the political bands which have connected them with another and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them "?
Or do they not?