White feminie virtue was esteemed. This is why so many slaveowners had relations (whether or not against the will of) with their female slaves. Yeah, great time. People could talk a good game then.
The vast majority of white men, even in the South, weren't slaveowners.