That's a misleading statistic. Almost all of those 350,000 slave owners had spouses, so that means up to 700,000 people directly benefited from slavery. Say each of those families had three children and suddenly you are up to 1.8 million people people drew direct benefit from slavery. That's large enough percentage of your population right there to understand why they may rebel to protect their slave property.
Of course that assumes Churchill was unaware of the problem you note. He was a good enough historian that I decided he would not make that mistake.
Families who own slaves is one reason I spoke above of "households". "Though it's undeniable that the Confederacy gave special status to slavery, records show that most Confederate soldiers came from households that didn't own slaves." States varied in how many families had slaves, but they were in the minority in every state. Wikipedia:
"One estimate is that in 1860, about 25% of households and 5% of the population (384,000 people) in the South owned at least one slave. An alternative estimate is that 36% of men lived in slaveholding families, and the percentage of men who had economic ties to slavery was much higher."
Still most didn't.