Most likely friends and relatives (remember Elizabeth?) kept an eye on the children, so that Mary and Joseph could hurry back and search.
The fact that they went so far, before noticing that Jesus was not with them, would seem to indicate that they were quite occupied with small children.
Mark’s account, of the Lord preaching to his home town, shows that the locals were stunned to see how different Jesus was from the rest of his family. They had known the whole family for years.
So you think the whole family forgot about Jesus until they got home? This sounds like an episode from Everybody Loves Raymond.
It sounds like an error from cultural bias to me; why would a mother with young children leave them when her husband could search for the young man with other male relatives ? He was no longer a baby and it was a patriarchal society. And there is that word syngeneusin ... translated as relatives in this case, while sometimes as cousins; cousins = relatives ...
Now his parents went to Jerusalem every year at the feast of the passover. And when he was twelve years old, they went up to Jerusalem after the custom of the feast. And when they had fulfilled the days, as they returned, the child Jesus tarried behind in Jerusalem; and Joseph and his mother knew not of it. But they, supposing him to have been in the company, went a day's journey; and they sought him among their kinsfolk and acquaintance. And when they found him not, they turned back again to Jerusalem, seeking him.
Luke, Catholic chapter two, Protestant verses forty one to forty five,
as authorized, but not authored, by King James,
boldness mine