I grew up in a large family, 8 kids. My dad worked and my mom didn't drive so while we had a car, it was registered and licensed in my father's name and he was the sole driver. So one way of looking at it is that only 10% of the people in my family owned a car. But we all reaped the benefits of car ownership. Likewise with slavery. The total number of slave owners may have been around 6 or 7 percent, depending on the state, but undoubtably almost all of those slave-owners had wives and children. Looking at it that way, almost 50% of the families in Mississippi owned slaves. Over 40% of the families in South Carolina and Georgia. Over all, almost 30% of the families in the south owned slaves. So that is why it was so important to them. They weren't fighting to defend the rights of 6% of the population, they were fighting to defend an institution that almost a third of them had directly benefited from.