“No, it was the Confederates who fought for the right of their white people to buy and sell their black people like cattle.”
See our present condition-—the country engaged in war! Our White men cutting one another’s throats! And then consider what we know to be the truth. But for your race among us there could not be war, although many men engaged on either side do not care for you one way or another. Why should the people of your race be colonized, and where? Why should they leave this country? This is, perhaps, the first question for proper consideration. You and we are different races. We have between us a broader difference than exists between almost any other two races. Whether it is right or wrong I need not discuss, but this physical difference is a great disadvantage to us both, as I think your race suffer very greatly, many of them by living among us, while ours suffer from your presence. In a word, we suffer on each side. If this be admitted, it affords a reason at least why we should be separated. It is better for both, therefore, to be separated.”
— Spoken at the White House to a group of black community leaders, August 14th, 1862, from COLLECTED WORKS OF ABRAHAM LINCOLN, Vol 5, page 371
That doesn’t dispute what Lurking Libertarian said.