Except that the Lord of the Manor didn't buy and sell his workforce.
Many of their own free will, freed their slaves, giving them, or allowing them to purchase land, along with the means to start their own lives.
Comparatively few did. And many if not most Southern states had laws severely restricting the ability of the slave owner to free his chattel. And those who were free were not and could never be citizens or enjoy any rights, thanks to Chief Justice Taney. Another difference between the Southern U.S. and the English manor lords.
Otherwise just because one owned a slave did not mean he thought the black person was a lower form of life.
Oh please.
The remarks you have quoted "as racist" could be referring a former slaves' limitations due to the system of that time, not his skin color.
Then by all means let's see some quotes from Davis or Lee or Jackson or any other Southern leader of the period before or during the rebellion who believed the black man was his equal in any respect at all.