The vast vast majority of southern (& northern) whites had nothing whatsoever to do with slavery.
Up North this is true. Slavery was illegal in most of the Northern states.
In the South, a different Story. Based on the 1860 census.
Free white families owning slaves as a percentage of the States population: MS 49%, SC 45%, GA 37%, AL 35%, FL 34%,
LA 29%, TX 28%, VA 26%, TN 25%, KY 23%, AR 20% MO, 12%,
MD 12%, DE 3%.
The number of slave owners in the South was of such a large number that they had a virtual lock on Southern political power. In the border states, MO, KY, MD, and DE, slave ownership is much lower and the slave owners do not have the political power their counterparts in the deeper South have.
If true, I stand corrected.
Link to those figures ?