He didn't have him over for Christmas punch in 1865. Nascent abolitionist sentiment was fairly widespread in the South by 1860. Not dominant, not nearly a plurality, but the snowball had begun to roll down the hill.
The real unasked question in all this is where did antislavery sentiment come from? The answer was British social Christianity, the origin of Methodism, the Salvation Army, temperance and abolitionism. The British sent warships to disrupt the slave trade out of religous sentiment, not for imperial or commercial gain. That slavery was immoral, "wrong", had never been so strongly espoused or defended as in Victorian England and New England.
Well, that's unarguable since Clay died in the early '50s. But as you state, the South was not united in favor of slavery by any means. It was the large land owners who owned slaves, and they had the greatest political clout.
But, the anti-slavery sentiment was growing in Europe as well as the North by 1860. And, as you state, even in the South. In fact, it was the predominant reason England, and to a lesser extent France, declined to recognize the Confederacy, even though it meant a blow to their textile industry during the war.
Exactly backwards historically.
In the late 18th and early 19th centuries it was, among educated men at least, accepted nearly universally throughout the South that slavery was evil and must ultimately be gotten rid of.
By the 1850s this had completely turned around and slavery was (almost) universally accepted as a positive good that should not only be preserved but expanded. In fact, the newly elected veep of the CSA made this the main point of his most famous bit of rhetoric, the famous Cornerstone Speech. In it he explained why Jefferson, Washington and all those earlier southerners who had believed slavery to be immoral were wrong. Human inequality, in its ultimate form - slavery, was to the the Cornerstone of the CSA, rejecting and replacing the USA's Cornerstone of human equality expressed in the Declaration of Independence.
If you accept, as Lincoln did and I do, the DOI as the best expression of what it means to be an American, the Cornerstone Speech proclaimed a declaration of war by the CSA on the very concept of America.
Stephens' speech was widely plauded throughout the South. In fact, I have never come across a single expression of disagreement with its basic themes in any contemporary southern document. IOW, he expressed the conventional wisdom of his society.