If you are married and you deny your spouse sex, do you not consider that wrong?
God does.
1 Corinthians 7:3-5 The husband should give to his wife her conjugal rights, and likewise the wife to her husband. For the wife does not have authority over her own body, but the husband does. Likewise the husband does not have authority over his own body, but the wife does. Do not deprive one another, except perhaps by agreement for a limited time, that you may devote yourselves to prayer; but then come together again, so that Satan may not tempt you because of your lack of self-control.
Ridiculous that you interpret this as having to do with Mary.
There is no evidence that Paul had Mary and Joseph in mind when he wrote 1 Corinthians 7:3-5. Moreover, Paul permits abstention from marital rights by mutual consent in 7:5.