His thinking on the matter of racial equality evolved over time. Toward the end of his life he became much less of the white supremacist. IMHO, it was likely because the valor of black soldiers during the war impressed him.
Even Northern whites who rarely saw a black person worried about what would happen if the slaves were all freed and came North to compete for jobs—and most Northern states did not let free black men vote.
That changed with the ratification of the 15th Amendement.
That's why the 15th amendment was needed--Congress had already forced the former Confederate states to extend the franchise to black men, but some of the Northern states still restricted the vote to whites. After the 1868 election the Republicans realized that the small number of potential black voters in various Northern states could make the difference in a close election determining who won the electoral votes (and they could figure on all or nearly all of those votes being cast for the Republican candidate).