Properly speaking, he was doing a bit more than "visiting" Egypt. He was taken there by Joseph and Mary to escape Herod's murderous designs.
He was taking political/religious asylum in Egypt, to be exact.