I found some additional information about how Floyd and Buchanan reacted to Anderson's move into Sumter in the book, "Days of Defiance," by Maury Klein.
...The next morning when Trescot was readying their credentials [the credentials of the South Carolina commissioners who had come to talk with Buchanan], Louis Wigfall burst in with a telegram that Anderson had spiked Moultrie's guns and moved to Sumter/ The commissioners and Trescot were stunned. "True or not," said Trescot amid an animated discussion, "I will pledge my life that if it has been done it has been without orders from Washington.Just then Floyd arrived. He blanched at the news and confirmed what Trescot had said, that such a move "would be not only against orders but in the face of orders." ...
Trescot informed Senators Jefferson Davis and Robert Hunter and went with them to the White House to demand an explanation. Buchanan had not heard of Anderson's action. Here is how Buchanan responded according to Klein's extensively documented book:
Buchanan slumped into a chair. "My God!" he cried wearily. "Are calamities ... never to come singly! I call God to witness -- you gentlemen better than anybody else know that this is not only without but against my orders. It is against my policy." ...
Here is a communication from Captain J.G. Foster, one of the engineering officers in Charleston describing the work being done at Forts Moultrie and Sumter.
Colonel R. E. DE RUSSY,
Commanding Corps of Engineers, U. S. Army, Washington, D. C.:
COLONEL:
...
I am abating nothing of the activity of preparation in Fort Moultrie and Fort Sumter, and in fact am increasing it.
If the Department becomes aware of any change of policy in regard to this preparation in these forts, or in either of them, I beg that instructions may be given me at once, so that I may vary my operations accordingly, for my p[resent expenses are very heavy. In Fort Sumter the mounting of the guns, laying a flagging of first and second tiers of casemates, forming embrasures of second tier, and finishing the barracks is progressing regularly, and as fast as separately organized parties can work. The force will be to-morrow 150 men.
...
J. G. FOSTER,
Captain, Engineers.
DECEMBER 19, 1860.
I think the answer may well be that the number of civilians working at Ft. Sumter varied over time, but that in the days leading up to Major Anderson moving his garrison there, a sizable workforce was employed.
Enjoy your visit to Sumter.