The claim that "only about 5% of whites owned slaves in the South," is true, in a sense, but also a canard.
According to the 1860 census, there were about 8 million whites in slave owning states. Of those, 385,000 owned slaves -- or about 5%. But these were all relatively wealthy heads-of-households, meaning they also had large families.
If the average immediate family size was, say, six (a wife and four children), now we see that about 30% of southern whites lived in the homes of slave holding families.
And consider a typical young white farming family in the Deep South. If they did not themselves own slaves, their parents did, and the young family would too -- as soon as they could afford them. So they were in no sense "anti-slave."
And that is just "average" for the entire South, where the slave population ranged from well over 50% in Deep South states like South Carolina and Mississippi, to about 25% in Upper South states like Virginia and Tennessee, to barely 10% in Border States like Maryland and Missouri.
So, while the average of slave-owning white families may have been 30% overall, the range was from well over 50% in the Deep South to under 10% in Border States.
Indeed, in states like (western) Virginia, (eastern) Tennessee and (western) North Carolina, many counties of the state had relatively few slave owning households and so refused to seceed, or resisted secession. Some of these also supplied soldiers for the Union army.