If the catholic claim of Mary's perpetual virginity is true she entered the marriage under false pretenses as outlined by the pope which would render their marriage null if we're understanding this claim.
God is all-honorable and all-just. He doesn't "do" adultery.
Therefore though Mary was in one sense of the word, "legally" married to Joseph, we can safely presuppose that both Mary and Joseph knew she was not married married to him "understood in the normal sense" --- and therefore she was not lying to Joseph nor guilty of marital fraud towards him. A mutual vow of chastity (I presume that here you actually mean abstinence) would not be false if Joseph had known and agreed.
If he did NOT know, then, yes, it would be marital fraud on the part of Mary; and God would be committing adultery.
This is why the Muslims do not believe in the Gospel account of the Incarnation. They say it casts God as an adulterer.
I think the Muslims are wrong of course. But they have a logical point: if one thinks Mary had a "normal" marriage agreement with Joseph, you have to contemplate the "divine adultery" of God impregnating somebody else's wife. Which is, I would say, a blasphemous thought.
Not to worry: that is only a valid basis for annulment if the pope did not grant a dispensation, who is as God, and thus since Mary had God's sanction then it was as good as the pope's. But David and Abishag the Shunammite is another case, married or not.