If the characterization of Tertullian is found to be correct then yes, I will admit that it is not true to say that "everyone" believed it. So we're down to "almost everyone".
I'm just not sure how we get from "almost everyone accepted it" to how it's being treated here--like it's an abominable heresy.
Will you concede, at least, the ever-virginity of Mary is a theological proposition that may be held by a Christian? That it is not heretical?
Paul predates your traditions. HE spoke of the brother of Jesus. That is sufficient for me. Your mileage will be gauged by the religious institution and its man-made traditions which you favor over The Word of God.