Well to be fair one of the wise men did tell Joseph and Mary to flee from Bethlehem to Egypt to escape Herod’s killing of the innocents.
“Well to be fair one of the wise men did tell Joseph and Mary to flee from Bethlehem to Egypt to escape Herods killing of the innocents.”
..and they went to Bethlehem to be counted so that they might be taxed.
They came back but I suppose for a brief period he was, indeed, a refugee or, rather, his parents were. He went along for the ride.
An angel told them to flee to Egypt, not one of the wise men.