I think that applies to the BAD and CRUEL slave owners, of which most were not.
Yes, there were some slaveowners who tried to treat their slaves decently, and others who were cruel, but from the slaves' point of view it was all bad. Man is not meant to be a slave. See the autobiography of Frederick Douglass. He differentiates among the people he served, and gives credit to those who treated him more fairly or decently, but found plenty to be discontented with even while serving the best masters.
I wouldn't disagree that many slave owners were not cruel. But who among us would want to submit to slavery on the likelihood that we would get a kind owner?
Men weren't meant to be property- not even of a kind owner.