I certainly agree with you on this. There was a great divide between the elites and the common folks in both the North and the South regarding justifications for fighting the war. And many memoirs of soldiers in every war reveal that their foremost concern is the safety and well-being of their comrades-in-arms, not some abstract ideology or moral cause.
Interestingly, there were more free blacks residing in the South in 1860 than in the North. Of course, almost all of the blacks in the North were free, and only about 7 percent in the South were free. Any the vast majority of newly free blacks in the South after 1865 remained there, many continuing to work for their former masters as wage labor.
Actually most were sharecroppers. And a generation or two later they would flee the south by the millions.