The sacrifice of turtle doves always seemed incongruous for a family that otherwise seems fairly well-to-do. Perhaps they were cleaned out by the taxes they had to pay in Bethleham.
Or just hadn't made it back home between the birth and the Purification. They probably buried their money in the kitchen floor, if they had any actual coinage.
I don't think that everyone in the family tree was rich. I think if Joseph was rich, he could have taken a whole herd of animals and also the people that worked for him to Bethlehem. They could have build a shelter for Mary to have her baby in.