I think saying “It would have been better to have left them alone” is more demeaning, since the “them” to whom you refer is people who owned other people as property.
Them refers to the enslaved Africans and the owners in the South. It’s pretty obvious the owners needed the labor to acquire the wherewithal to provide all with food and shelter so eventually they’d work out a deal. At any rate, being an enslaved African in the South was considerably preferable to being an enslaved African in Africa, Arabia, the Caribbeans and virtually everywhere else. It was even preferable to being a Russian peasant or a European serf.