I don't think that everyone in the family tree was rich. I think if Joseph was rich, he could have taken a whole herd of animals and also the people that worked for him to Bethlehem. They could have build a shelter for Mary to have her baby in.
I didn't say rich. He wasn't like Barbra Streisand or Jesse Jackson, but it seems that he was solidly middle class, equivalent to a contractor or cabinet maker in our time.