Jesus grew up in Nazareth, and Joseph was dead when the ministry began. So it would be easy to cast aspersions on his lineage.
Some Protestants think that the Lukan genealogy is of Mary. The Fathers are unified that this is not the case.
Neither Jew nor Greek would have considered a female lineage to be valid.
How do you know Joseph was dead? Scripture doesn't say he died. Maybe he was assumed into Heaven.
I of course am being facetious.