The very small percentage of white slaveowners you have been promoting is probably based on the percentage that owned “more than 20” or “more than 50” slaves. The wealthy planter was indeed a small percentage of slaveowners and of white families.
But there were a great many white families that owned one or a few slaves. Most of them worked alongside their slaves in the fields. Could not afford to do otherwise.
Check out the actual 1860 Census records. I think you’ll find them enlightening.
Instead of assuming and saying “probably” read the articles I posted and you will find that you are incorrect. Just do a search of “what percentage of blacks owned slaves” and you might find more that will enlighten you.