One of the reasons Lincoln wanted to keep slavery from the territories was to protect the opportunities of free white workers (another was to decrease opportunities for miscegenation). Speaking at Kalamazoo, Michigan, in 1856, he said that the territories “should be kept open for the homes of free white people.” Even his cherished plan of sending freed blacks to Liberia was looked at from the economic vantage of free white labor. In his 1862 annual address to Congress, he said: “With deportation, even to a limited extent, enhanced wages to white labor is mathematically certain.”
Slavery not only diminished the white worker’s economic equality, it eroded his political equality. The constitutional provision by which the slave states counted blacks as three fifths of a person in the census meant that “three slaves are counted as two people” in Congress, with the result that “in all the free States no white man is the equal of the white man of the slave States.” Lincoln repeatedly argued against slavery as violating the interest of white workers. This is what Frederick Douglass meant in 1876, when he said of Lincoln:
He was preëminently the white man’s President, entirely devoted to the welfare of white men…. He came into the Presidential chair upon one principle alone, namely, opposition to the extension of slavery. His arguments in furtherance of this policy had their motive and mainspring in his patriotic devotion to the interests of his own race
Let's look at the Frederick Douglass speech, though. Here's what he also said:
His great mission was to accomplish two things: first, to save his country from dismemberment and ruin; and, second, to free his country from the great crime of slavery. To do one or the other, or both, he must have the earnest sympathy and the powerful cooperation of his loyal fellow-countrymen. Without this primary and essential condition to success his efforts must have been vain and utterly fruitless. Had he put the abolition of slavery before the salvation of the Union, he would have inevitably driven from him a powerful class of the American people and rendered resistance to rebellion impossible. Viewed from the genuine abolition ground, Mr. Lincoln seemed tardy, cold, dull, and indifferent; but measuring him by the sentiment of his country, a sentiment he was bound as a statesman to consult, he was swift, zealous, radical, and determined.
What do you think Douglass meant a few lines later when he said this?
"When, therefore, it shall be asked what we have to do with the memory of Abraham Lincoln, or what Abraham Lincoln had to do with us, the answer is ready, full, and complete. Though he loved Caesar less than Rome, though the Union was more to him than our freedom or our future, under his wise and beneficent rule we saw ourselves gradually lifted from the depths of slavery to the heights of liberty and manhood; under his wise and beneficent rule, and by measures approved and vigorously pressed by him, we saw that the handwriting of ages, in the form of prejudice and proscription, was rapidly fading away from the face of our whole country; under his rule, and in due time, about as soon after all as the country could tolerate the strange spectacle, we saw our brave sons and brothers laying off the rags of bondage, and being clothed all over in the blue uniforms of the soldiers of the United States; under his rule we saw two hundred thousand of our dark and dusky people responding to the call of Abraham Lincoln, and with muskets on their shoulders, and eagles on their buttons, timing their high footsteps to liberty and union under the national flag; under his rule we saw the independence of the black republic of Haiti, the special object of slave-holding aversion and horror, fully recognized, and her minister, a colored gentleman, duly received here in the city of Washington; under his rule we saw the internal slave-trade, which so long disgraced the nation, abolished, and slavery abolished in the District of Columbia; under his rule we saw for the first time the law enforced against the foreign slave trade, and the first slave-trader hanged like any other pirate or murderer; under his rule, assisted by the greatest captain of our age, and his inspiration, we saw the Confederate States, based upon the idea that our race must be slaves, and slaves forever, battered to pieces and scattered to the four winds; under his rule, and in the fullness of time, we saw Abraham Lincoln, after giving the slave-holders three months' grace in which to save their hateful slave system, penning the immortal paper, which, though special in its language, was general in its principles and effect, making slavery forever impossible in the United States. Though we waited long, we saw all this and more."
Or this in closing?
"Fellow-citizens, I end, as I began, with congratulations. We have done a good work for our race today. In doing honor to the memory of our friend and liberator, we have been doing highest honors to ourselves and those who come after us; we have been fastening ourselves to a name and fame imperishable and immortal; we have also been defending ourselves from a blighting scandal. When now it shall be said that the colored man is soulless, that he has no appreciation of benefits or benefactors; when the foul reproach of ingratitude is hurled at us, and it is attempted to scourge us beyond the range of human brotherhood, we may calmly point to the monument we have this day erected to the memory of Abraham Lincoln."