Poor southerners who backed slavery...I believe the proper term for that is Democrats.
Slavery was a system of labor which brought profit to the slaveowners, but it was also a system of racial control—”white supremacy”—which most whites believed in. Lincoln abhorred slavery, but at least in his public comments supported the idea of the white people being on top in society. Even Northern whites who rarely saw a black person worried about what would happen if the slaves were all freed and came North to compete for jobs—and most Northern states did not let free black men vote.