Joseph was not her husband and all the children they had were not theirs, they were cousins. Or so says the gospel of Mary.
;-)