They were also pretty rich from the Magi gifts, which supported them during the exile.
Joseph was an artisan and the father of a large family. They were a stable, well off family. Jesus did not come from poverty.