Are you saying only people of one race pick cotton? There were many poor whites who also picked cotton on small farms. So, there is no racial component. Cotton picking is a back-bracking business. It’s a great metaphor. Leave it alone.
My now 80-year-old (white) mother picked cotton as a child in Texas. She always told me about how heavy the long sack was and how the bolls pricked her fingers.