Unlike the north, the South didn't segregate blacks and whites into racially divided units.
Segregation didn't exist in the South, in it's northern institutionalized version, until after Redestruction.
I am sure you are right, that the blacks lived in the big house, and the whites lived in the slave cabins often as not.
I am also sure, since race didn't matter, that blacks owned white slaved.
Right.