Reminds me of the claim that Peter never married yet scripture speaks of Jesus healing Peter's "mother-in-law!"
Kind of hard to do when you're not married, isn't it.
Scripture speaks of other "brothers" to Jesus, who would be half-brothers and actual sons of Joseph and Mary.
It's tough to let go of tradition from early fathers but to be credible, in an age where bibles abound everywhere and even more than one copy owned by many people.......and read every day and in complete each and every year!
This is probably why the early martyrs died to bring the scriptures to the people and why the church worked so hard in the other way.
The bible said what some people said it said, now we all can read it ourselves, and with the help of the Holy Spirit, interpret the meaning without the help of some clergy, if we desire.