You asked “why not — why did Joseph refrain from having sexual relations with Mary,..”
Who knows? The point of the way it played out was that Joseph knew HE DID NOT have relations with Mary.
So he wanted to divorce her until God set him straight.
Hebrews were allowed conjugal visits during the betrothal period.
Again, God would not subject Mary to scandal by impregnating her during this period of time. He would not reign down a perception of sin upon her!
As to why Joseph did NOT have sex with her? Who knows? It doesn’t detract from the Virgin birth - the fact that he knew he didn’t have relations with her made the Virgin Birth more clear...
The larger question is why her community did not try to stone her for being pregnant. They didn’t because conjugal visits were allowed.
Find a scripture that says that.
Here is a scripture that says sex is not permitted during betrothal.
Paul said (2 Cor. 11:2) “For I have betrothed you to one husband, that I may present you as a chaste virgin to Christ.”
The Church is the Bride of Christ, but the Wedding Supper is clearly spelled out in both parables and in eschatological scripture.
The Church and Christ are fully committed during the betrothal period, but the consummation has not yet taken place.
Most modern commentators, following the footsteps of St. Thomas, understand that, at the epoch of the Annunciation, the Blessed Virgin was only affianced to Joseph; as St. Thomas notices, this interpretation suits better all the evangelical data. ...The Incarnation
This marriage, true and complete, was, in the intention of the spouses, to be virgin marriage (cf. St. Augustine, "De cons. Evang.", II, i in P.L. XXXIV, 1071-72; "Cont. Julian.", V, xii, 45 in P.L.. XLIV, 810; St. Thomas, III:28; III:29:2). But soon was the faith of Joseph in his spouse to be sorely tried: she was with child. However painful the discovery must have been for him, unaware as he was of the mystery of the Incarnation, his delicate feelings forbade him to defame his affianced, and he resolved "to put her away privately; but while he thought on these things, behold the angel of the Lord appeared to him in his sleep, saying: Joseph, son of David, fear not to take unto thee Mary thy wife, for that which is conceived in her, is of the Holy Ghost. . . And Joseph, rising from his sleep, did as the angel of the Lord had commanded him, and took unto him his wife" (Matthew 1:19, 20, 24).