Well, in the loaves and fishes episode, Matthews says there were about 5,000 men, as well as women and children. Maybe it would have been socially inappropriate for the Apostles to have asked other men’s wives and daughters to serve and clear up! (Or maybe they did, but Matthew doesn’t mention it.)
It's just another one of those unanswered questions we might have.
I think that YOUR answer of being "socially inappropriate" hit the nail on the head.