Pure(?) speculation.
If Joseph had other older children, why didn’t they go with him and Mary to the mandatory census in Bethlehem?