Also, while Egypt and Judea were separate countries, they were both part of the Roman Empire so really it was like Joseph, Mary, and Jesus fled New Mexico and escaped into Texas.
In other words, are you really a refugee if you leave one part of the Roman Empire only to find refuge in another part of the Roman Empire?
Separate countries, separate people with separate cultures and languages.