Very good point. Sometimes I think, given the essential support of the Free-soiler movement in the Old Northwest (which included Mostly Honest Abe), that the anti-slavery movement had about it something of a Full Employment Act, given the tendency of slave labor in antiquity and in the South to create economic inequality and drive freehold farmers off the land.
You can't make a case in finance or economics against bond slavery: what the world needs, every industrialist and planter alike would have told you 140 years ago, is a good five-cent human being.
Hence the Free-soilers were correct to worry about the possible introduction of cotton-resuscitated, slave-powered agribusiness into areas where they wanted to go, like the Nebraska Territory. In Texas, historian Fehrenbach has recorded in Lone Star, his manual of Texas history, slaveowners like Jared Groce, who accompanied the Old Three Hundred to Texas in 1820 and 1821, were able to use their bondmen to stake out much more acreage than the freeholders under the Spanish land-grant rules. Each slave was allowed to stake out a holding, which then inured to his master -- how that worked, and why the Spanish sat still for it, is a minor mystery to me still. But that is how the planters in Texas, whether accepting mercedes from the Spanish Crown or, later, staking claims under the Republic, were able to put together very large parcels of "peach-bottom" or prize bottom-lands, and achieve the economies of scale that the freeholders both North and South rightly feared. Only in the South, the wrath of the freehold, hardscrabble farmer turned against the black bondman, not the Man in the Big House.
So it seemed, all through the ages. But an economic case against slavery really can be made. Toqueville made it. In his splendid "Democracy in America," near the end of the book he is taking a slow boat-ride down the Ohio River. He described what he saw along the Kentucky bank and on the Ohio bank.
Ohio was hustling and bustling. Kentucky was languid and pastoral. Toqueville observed that on each bank of the river there existed the same climate, same soil, same people, same language, same religion, same everyting -- but Kentucky had slavery. He brilliantly concluded that in Ohio, work was honorable, and men were out there, building and hauling and getting things done; while in Kentucky, work was that "they" did, and gentlemen were supposed to follow other persuits (hunting, leisure, etc.), so Ohio prospered while Kentucky stagnated.
Reading Toqueville was the first time I realized that not only was slavery bad for "them", but it was bad for "us" too.