Its Hypocritical to state that only the south had a caste system. The caste system was very strong in the northeast: There were the bankers and corporate barons, followed by the merchants, then the lowly recent-immigrants and indentured servants. The recent immigrants and their children worked long hours in sweat shops and other labor intensive jobs. Later, Chinese immigrants and their families were used as low-wage laborers in the far west. Native Americans were being pushed westward.
The stratification in the south was bad as well. There were a very few but highly-influential plantation barons, a sizeable population with only a few slaves, a significant group of poor whites with no slaves, native Americans, and of course the unfortunate slaves. Even the slaves had their own caste system.
I didn’t say that only the south had a caste system.