That's primarily because of interracial relations with the white community (with a few other factors coming into play), but not at all because of geography.
Some Blacks I have seen kinda look more Caucasian to me, especially their hair, which seems to more straight. Also, their facial features are more Caucasian-like.
If you were lactose intolerant, and had darker skin, life in a Northern city such as Chicago might well leave you somewhat incapacitated as compared to your lighter brethren who were not lactose intolerant.
Not that this couldn't be overcome in these modern times, but unless your mother was fully cognizant of what was happening, she could end up with many fewer grandchildren than she might have had in Mississippi for example.
Delection of certain genetic combinations can happen in a single generation if they prove fatal, or nearly so.