Good point. I started reading a book by Anne Rice (yes, that Anne Rice) that is a fictionalized story of the childhood of Jesus. It painted a pretty good picture of life back then. It didn't show them as dirt poor but because Joseph was a carpenter he had good skills to provide for his family.
If Joseph was a “carpenter,” then he was a a craftsman, which put him among the middling sort. The tradition that has Mary coming from a priestly family living near Jerusalem might mean that her family as well as Joseph had recently migrated to Nazareth, following other Jews who had come north earlier. Fact is that we know so little for sure.