Too busy caring for all those other kids to keep track of him.
Though there was NO MENTION of these other children when they were at the Temple (and remember they would ALL be under 12 years of age).