I see that point, and it makes sense to a degree. However, the same points could be made to the situation after slavery when most of the slaves became sharecroppers. It would seem to me the “widespread benefit” was really only to the large landowners and their families who benefited from cheap labor before and after slavery.
Well, sure. The people who most directly benefited from cheap black labor after the war were mainly the same people who benefited from slave labor before the war. But I'd argue that it's not just the planters, but the middle class who owned a slave or two as household help and continued to employ a housekeeper and a man to work around the property. But in an agricultural society, which the south was until fairly recently, the secondary benefits extend throughout the society--the bankers, the merchants, the professionals and so on--the people who take the money that the planters spend.
Bringing it back to the Civil War, though, lets say that a landowner has 20 slaves and five sons. All of those sons have a huge stake in maintaining slavery--it's fundamental to the family wealth and those slaves are probably worth more than the land they work. Are any of them on the books as slave owners? No. But it would be deceptive to say that because they're not slave owners their presence in the confederate army shows that slavery had nothing to do with their reasons to fight.