When I look around at Jews I see every day walking to services on Saturday or around downtown, I see a lot of pale skinned people with red and brown hair.
These typical dark-skinned middle easterners are a product of centuries of intermixture of Arabs (meaning people from the Arabian peninsula), African slaves, and actual middle easterners.
If you look at the people in the middle east who have tended to keep themselves apart from the Arab/Muslim intermixture - Lebanese and Egyptian Christians for example, you see pale skinned people who frequently have light hair and eyes. Same thing goes for depictions of them in ancient frescoes and in Egpytian Heiroglyphic scenes. Jews have also tended not to intermix in their exile.
Ps., the Egypt Mary and Joseph hid out in was most likely cosmopolitan Alexandria, which would have had quite a number of Greeks and Romans, in addition to Phoenicians, Egyptians, and Jews. None of these people were renowned for being swarthy at that time.
You have pictures?