Racism was baked into the cake. The entire premise of slavery was that they were an inferior race and therefore there was nothing wrong with forcing them to serve.
But what your grandmother remembers is what actually happened. I think there would have been less animosity and hatred had slavery been abolished gradually.
For one thing, the Northern Liberals were using the blacks as a tool to keep them in power in Congress, and the laws they passed and the people who were appointed to govern the South, allowed them to be abused and exploited in a manner that wouldn't have happened without the war.
The blacks were seen as helping the enemy; as the tool of the enemy. I think in an alternate timeline of gradual emancipation, the racism would have still been there, but the hatred would have been less. They would have been seen more as harmless, than as a means of helping the oppressive government.
Well, it happened the way it happened.