Yes, there were big differences in the three classes in the South, and people like Longstreet weren't interested in preserving slavery.
That seems to be a very uncomfofrable fact for those who buy into the myth of the rebellion being a pure movement arising from the southern people. The rebellion was the result of political manipulations from a decaying ruling class in the interest of slavery. I believe the ignominious collapse of the reb armies testify to the illegitimacy of the movement. The Confederate soldiers in the end were realizing that a Slave Empire was not worth the price.