Yes, I’ve heard that. I never lived down South, but have heard, that black and white both worked the fields. But mostly we hear of the legacy of slavery, and black sharecroppers struggling to earn a living, and then they create caricatures of slaves or black sharecroppers toiling away.
My fiancee’s parents are native Alabamians, they all worked in cotton fields as kids and this is in the 50’s and 60’s.