You have made a very important point. Like you, I couldn't care less what Mary did after the virgin birth of Jesus Christ. She had every right and it was totally within the law that she have a normal husband/wife relationship and have other children. She was certainly a very blessed woman and, through her faith and obedience, she was given the great honor of bearing the Messiah. But that's it.
If the Catholic Church had left it at that or even if they had left it up to the individual to accept her continued virginity, I would have no problem. What they did that was without Scriptural authority was to make the belief in her perpetual virginity as well as her "sinlessness" a mandatory rule/law of the faith. They proclaimed that it was of the faith and that those who denied it could NOT be saved. If anyone rejected it they were told they were self-excommunicated. In other words, believe what we say is true or you will go to HELL. They went TOO FAR. That is why I reject their assumed infallibility and why I know they are NOT teaching the truth. "Do not go beyond what is written", is what Paul said. They did just that and not only on this point, I may add.
I think that may be misunderstood.