It was the confederate leadership that took the average citizen or soldier to war to begin with.
The average citizen voted to secede. I don’t believe they voted to secede to enable a small group to keep their slaves. They voted to secede, because they didn’t like other people interfering in their business.