so the slavers really called their executive mansion the “White House?”
I think it was Theodore Roosevelt who first used the term "White House" to officially refer to the Executive Mansion. I would assume that referring to the Confederate version as the White House is more recent as well. Actually, the building in Montgomery that is usually referred to as the first Confederate White House was painted grey.