The White House was pretty much totally rebuilt in the early 1950s. The Trumans had to move across the street into Blair House.
During the Clinton administration, Hillary tried doing her own Jackie Kennedy type White House tour. She told the reporter that she was moved thinking about walking the same halls that Lincoln walked.
The White House was gutted down to the exterior shell, and even that has been replaced now with improvements made during George W. Bush's admin and Obama's admin.
Nothing left that was built by slaves, unless they made some of the furniture that survives.