In the slave-owning states, it was possible for middling folk to set themselves up as "lords and ladies" so to speak. A sort of gentry class that they never otherwise would have attained had they not had slaves to do all the domestic and farm chores for them.
For them, the ending of slavery would mean that they would lose that status and become mere employers at best. In fact, most of them would need to reduce "staff" significantly and take on much of the actual work themselves. The ladies would have to get off their couches and spend time in the kitchens. The gentleman would need to get off their front porch rockers, roll up their sleeves, and get their hands dirty for field and barn chores.
People don't like to talk about this aspect but the prospects of losing their status as "masters" and thus getting moved down the social ladder accordingly horrified many of them.