Slavery was indeed a caste system, but when it was abolished, that caste system didn't go away, it simply got applied to everyone else in the nation who was not part of the rich elite in the northeast.
You know, they same corrupt bastards still running Washington DC today.
As Randy Newman put it...
Now your northern n*****’s a negro
You see he’s got his dignity
Down here we’re too ignorant to realize
That the North has set the n***** free
Yes he’s free to be put in a cage
In Harlem in New York City
And he’s free to be put in a cage
On the South Side of Chicago
And the West Side
And he’s free to be put in a cage
In Hough in Cleveland
And he’s free to be put in a cage
In East St. Louis
And he’s free to be put in a cage
In Fillmore in San Francisco
And he’s free to be put in a cage
In Roxbury in Boston
They’re gatherin’ ‘em up from miles around
Keepin’ the n*****s down