He didn't go to Egypt until the 4th or 5th book ... lived with the uncle and aunt.
Ahhh,
OK, I didn't read the books in sequence and I thought she was pregnant in the second book and yeah, the kid didn't come in as a character until later books.
It's been YEARS since I read the early books.