I suppose those who thought that possible did not remotely comprehend the tactical situation. Sumter would be a nearly impregnable fortress in regards to an attack from the sea. But as we saw in April of 1861, it was pretty much defenseless against land-based batteries. It was reduced almost to rubble less than 40 hours from those land-based batteries. Fort Sumter was built to defend Charleston against a naval invasion, not to defend itself against an attack from Charleston.
On February 17, 1865, the Confederates left the fort. By the end, the fort was pretty much a pile of rubble. There are some pictures of the rubble in Link 1 and Link 2. Those old brick forts couldn't stand up to the newer canons of the time.