The Confederacy never surrendered Fort Sumter, but General William T. Sherman’s advance through South Carolina finally forced the Confederates to evacuate Charleston on February 17, 1865 and abandon Fort Sumter.
Yes, Confederate troops were pulled from Charleston on February 17, 1865, for potential use against Sherman's troops in North Carolina [Source: The Bombardment of Charleston, 1863-1865 by W. Chris Phelps]. On February 17th Sherman's troops were busy burning Columbia. Sherman's troops did not come to Charleston, but Gilmore's troops, who had been bombarding Charleston civilians for 18 months, did.
With Confederate troops gone, Gilmore's troops did what they did best and looted Charleston. From the Official Records:
Major General Gillmore (Union) to General Hatch (Union), South Carolina, Mar. 1, 1865: "I hear from all sides very discouraging accounts of the state of affairs in Charleston; that no restraint is put upon the soldiers; that they pilfer and rob houses at pleasure; that large quantities of valuable furniture, pictures, statuary, mirrors, &c., have mysteriously disappeared ..."
Even after that, the city was subjected to further looting by Northerners [Source: Loot Link]:
Northern "tourists," many of whom were members of Rev. Henry Ward Beecher's congregation from Brooklyn, New York, looted other material from both public and private repositories in Charleston in April 1865.