That’s right and I doubt the journey was *dangerous*.
They were with a crowd and traveling together. That’s why they did not miss Jesus for the day.
If it were just them and an only child, they NEVER would have not noticed that He was not with them.
If there were several little ones and Joseph died leaving Mary to care for them all, the custom in those times was for first cousins, aunts and uncles to take one or two into their household to raise them, usually the ones older than infants to two years old. That would give strength to the naming of the brothers and sisters who were later named as the children from the households of Mary and Joseph's relatives/cousins. I am focusing on this notion in one of the novels I'm working on, Letters To Tacitus.
My wife came from a large family. She was 12 when left behind at a rest area. Sleeping, everyone but her had bailed to use the bathroom. She awoke, found everyone gone, also went to the bathroom; however, somehow missing everyone else, who were piling back into the station wagon. They were miles down the road before somebody noticed she was gone!