Joseph had nothing to do with that union.
His fruitful union with Mary Came later.
Matthew 1:24
Then Joseph being raised from sleep did as the angel of the Lord had bidden him, and took unto him his wife:
25 And knew her not till she had brought forth her firstborn son: and he called his name JESUS.
Marriage is fruitful even when no biological offspring is produced. Joseph's marriage was validated, and, in fact, glorified, by his care of Our Lady and child Jesus.