Yes, indeed. You're absolutely right.
But, look at it from the standpoint of the slaveholders. These are people who had become addicted to slavery. Often, their families had for generations been supported by the labor they stole from their slaves. They became trapped in a culture of dependency and that's why we hear the explanation that they "knew no other way of life." From the nursery, they were raised and cared for by slaves.
The slaveholders were parasites totally dependent upon their slaves for economic and personal support. Many of them talked about the need to end slavery, but most of them could not conceive of how they might make their own way in the world without the support of their slaves. So, they got lazy and did nothing and then, when they felt that their parasitic lifestyle was being threatened by talk of abolition, they very desperately tried to declare a "secession" so that they could continue to use the machinery of their State governments to protect their indolent lifestyles. The slaveholders felt trapped. They felt that they had no choice, no other option. So, that's what they did.
Most Southerners now are strongly opposed to slavery. Most now are very grateful that Lincoln and the United States freed the slaves and many now are also very grateful that Lincoln and the United States freed the slaveholders from their parasitic addiction to slavery. Granted, some of the slaveholders couldn't make it on their own and their lives could not be reconstructed. However, most regained their self-respect and moved on.
The Civil War was really just a case of very tough love.