Yes I agree, but then why would Joseph seek to divorce her?
The bigger question is that God would not give the appearance of sin by having Mary impregnated, even by His spirit.
The betrothal period allowed for conjugal visits, otherwise Mary would have caused a situation that would have required her to be stoned to death.
A divorce was required in the betrothal period. They were not “unwed” in the eyes of the Lord, lest Jesus was made a bastard by the Holy Spirit!
Because betrothal was a real committment, but it still wasn’t quite married. If as some said, Joseph got “conjugal visits” during their betrothal, then that would be a direct attack on Mary’s virginity, wouldn’t it be?