It would depend on who you talked to. Abolition societies were well established in the North, but the average Northerner probably had no interest in ending slavery and no more liking for blacks than the southerners did.
Is it possible then that Lee's views regarding slavery were somewhat enlightened in comparison to most of the country's population?