Scottish masons erected the walls, but that was only part of the labor required to build the White House. Who dug the foundations and unloaded stone and brick at the site?
Much of the work was done by African-Americans, some free, many slaves, and that's been documented. James Hoban even owned three of the slaves.
More here.
this