I'm talking about when they had to flee to Egypt to escape King Herod. They were refugees then.
"Now when they [the Magi] had departed, behold, an angel of the Lord appeared to Joseph in a dream and said, “Rise, take the child and his mother, and flee to Egypt, and remain there till I tell you; for Herod is about to search for the child, to destroy him.” And he rose and took the child and his mother by night, and departed to Egypt, and remained there until the death of Herod. This was to fulfil what the Lord had spoken by the prophet, “Out of Egypt have I called my son.” (Matthew 2:13-15)
Oh, right. I was only thinking about their first trip.