Thank you for that response.
We surely cannot deny that you were right in correcting the doctrine about children of Mary, and you were right in rejecting the idea that any other offspring should come from the same virginal womb from which Christ was born according to the flesh. . . . For if they accept the doctrine on the authority of priests that Mary had a number of children, then they will strive with greater effort to destroy the truths of the faith.
Those fears need not be present within you or I.
As others mentioned, there is the Protoevangelium of James which may as well be of later origin than those whom rely upon it (even though it be known to contain historical impossibilities) whom wish to push it back to earlier origin. Whichever way, it still comes across as an imaginative prequel, written after the fact that the Synoptic Gospels must have had wide circulation.
Here's a thesis which is difficult to refute, for it's information is fairly thorough, and precise.
All the alleged history offered for Mary when she was just a young girl, and her alleged vows, comes from the Protoevangelium which is an apocryphal work at best...but one clever enough, and widely believed enough early on, to not land in the pile labeled "spurious" which it may very well belong.
Notice that Origin's initial claim of the idea, he himself sourced being from the Protoevangelium of James.
Others, following his lead, did much the same. It appears the idea was so much repeated. Some whom would seek to discredit the divinity of Christ, denying even the virgin birth itself, quoted the various sources speaking of Christ's brothers as "proof", so a body of apologetic sprung up which we see repeated almost verbatim today. The echoes from past defenses, based somewhat upon a false story (not written by James, it is most widely believed by historians) but which has other dimensions which much cloud discussion.
I apologize for not digging out this reference and reasonable explanation [at the link] previously, when I first saw this thread. I came upon it again here, while searching for the very article linked to.