Remember that there is considerable disparity in income between the whites in Fairfield (second wealthiest county in America) and New Haven counties, and the blacks and hispanics who live there. You're not going to get some white person from Darien, Greenwich or New Canaan to work in the fast food restaurants there. That leaves the blacks and the Hispanics. A little anecdote: I have been told several times by Greek restauranters and Korean grocery owners that they hire the Latinos because "they don't give us attitude and don't lift from the register" (exact quote from a Greek diner owner in Brooklyn).
I suspect that it has something to do with the fact that the local white kids have been raised to think that such a menial job is beneath them. Also we've become pretty lazy as a nation and don't expect our kids to work.