It’s not a “racist” remark, it’s a “Southern” expression.
I grew up in the 40’s and 50’s, in Georgia. The area where I lived was inundated with Cotton Fields. I never saw a single, black, person picking any of that cotton.
The pickers I remember were all white, and as I got older, I picked cotton to earn a little extra spending money.
When I was stationed in Maine, a bunch of us GI’s would work part time harvesting potatoes (”badaddas”), again, for extra money.
The operative color here, was GREEN, not Black...even back in slavery days.
There’s not a lot of black folks who even know what a cotton boll feels like, or recognizes the smell of boll weevil poison on a warm, summer night.
So, STFU already. Find something that actually MATTERS to get pissed about.
It’s a saying up in Michigan too. My Grandma and aunts used to say it. To white people.