No, it wasn’t just about slavery. But, slavery was the main cause.
I find no fault in your reasoning except for the potential alternative of slavery fading as it did in South America. Many slaves were skilled craftsman and not simply unitelligent field workers. Such a skilled workgroup could have played a part in industrializing the South. What alternative future would have come had the North not gained in Congress or Lincoln elected, is simply specultation. I envision that it would not have faded, but quite the reverse. But that is the realm of fiction writing.
I don’t think it likely that it would have faded as a matter of course. Government action was required to end it. I do believe that a peaceful abolition would have occurred before the end of the 19th century. The big stumbling block was the equity of personal estates that included slave property. Very few would have surrendered that wealth voluntarily, and their heirs were there to pressure them against it.