he is mentioning the flight to Egypt not the trip to Nazareth
The flight to Egypt was when Jesus was about 2 years of age. Herod demanded all (firstborn?) male children 2 years and under to be put to death. Many people confuse this with Joseph and Mary (pregnant with Jesus)going to Bethlehem for the required census.
But, even so, the argument that the family were illegal immigrants/migrants is not valid. Joseph and Mary would have had money (gold from one of the Magi) and Joseph was a carpenter and would have been able to establish himself as a tradesman who could earn his way. He wasn’t going to get an EBT card or WIC/SNAP for baby Jesus.