I seem to remember that most of the farm labor upstate and in South Jersey back in the 1980s was done by Carribbean immigrants and, to a lesser extend, Puerto Rican migrants. The last white American migrant farm laborer in these parts probably retired 100 years ago.
You may be right. I know more about the west coast, having lived in southeastern WA for awhile. They were very much from Mexico. We had an apple orchard in our backyard. The migrants let us have any fruit that fell to the ground, but they protected what was on the trees.