The northern states were far wealthier than the south during the slavery era. It's not a coincidence that the financial center of the US has been New York, not Atlanta or some other southern city. Other northern cities like Boston have always had more of the largest financial institutions than the southern ones.
Further, even attributing the south's wealth in the present day to slavery is utterly moronic. What things that the slaves built are still there? I guess a few historical buildings have been preserved.
No doubt many of the plantation mansions were built with slave labor. Then many of them were burned down by the Union troops.
Various plantation houses are now used as museums and wedding venues.