For most of human history, Northern Africa was much more closely tied to Europe than it was to the rest of the African continent. It wasn't until the two forces of Muhammed and later Charlemagne that people started thinking of North Africa as not being a part of Europe.
Liybans have the same incident for red-hair today as the Irish.