Article I Section 8 Clause 17:
To exercise exclusive Legislation in all Cases whatsoever, over such District (not exceeding ten Miles square) as may, by Cession of particular States, and the acceptance of Congress, become the Seat of the Government of the United States, and to exercise like Authority over all Places purchased by the Consent of the Legislature of the State in which the Same shall be, for the Erection of Forts, Magazines, Arsenals, dock-Yards, and other needful Buildings;The assumption is that the federal government purchased the land from the states for the buildings in your example. As such, those became the property of the federal government and the states cannot take them back or exercise control over them.
-PJ
The question is becoming inevitable, with the risk of California and the West Coast declaring secession. That was apparently one of the options being explored, if Trump had won in 2020.