If this was that big a worry, the economics of slavery would have killed it in a couple of decades with everyone realizing that it was more cost effective to pay them than to keep them slave.
It's easy to say that looking back over the last 140 years. But in 1861 there wasn't a single southern leader who believed that slavery was dying or didn't believe that their children and grandchildren would still be enjoying it.
Secondly, the death of slavery would have required societal changes as much as economic changes, maybe even more than economic changes. The majority of slave owners owned less than 5 slaves, and a large percentage of those were middle-class whites living in cities and towns who used the slaves for domestic help and not as plantation labor. Economics would have affected them less, they just wanted their maids or cooks or gardeners or grooms. Their view of slavery as an institution would have to have changed.