A tough one but quite possibly yes. Some of the more radical groups formed after the war may have never taken hold. There were some owners pre war who were tyrants but that wasn't as much a black white issue more than it was a slave owner issue. The owner could have been black, white, or whatever. A lot of the slaves were considered family to the owners. When the war ended many returned to where they had lived. Many of the owners were destitute themselves at that point.
I live in a county that was a Union strong hold myself. Yet in the 1950's the high school was bombed because of a hand full of persons who did not speak for the majority. It was over forced integration. I have never though segregation was right and I'm old enough that as a kid I remember it. My wife who moved from this area in her childhood to within an hours drive of Greeneville, MS remembers it but she got along with black kids. Actually a black family lived on their property. That was in the 1950's BTW.
But what gets me is how people make out that those who were for slavery -- sometimes very much in favor of slavery -- or indifferent to it were actually against slavery in the fullness of time because they or their children or grandchildren would "eventually" have abolished slavery.
Abolitionists and other opponents of slavery or its expansion, by contrast, get all the blame for what happened -- for making trouble and stirring things up. So we get told that Lincoln prolonged the existence of some form of slavery that those kindly Southerners would have done away with.
But really, things like slavery or segregation don't end on their own. Somebody has to take a stand against them to get things started. Whatever faults Lincoln or the abolitionists or the radical Republicans may have had they deserve more respect than they get from some people.