Maybe some of it.
The White House was completely gutted during the Truman administration with only the exterior walls surviving. I don't know how much of the masonry work was done by slaves, but I doubt it was much.
Exactly and only the centre part. The east and west wings were built in the 20th century with no slaves obviously.