Paul predates your traditions. HE spoke of the brother of Jesus. That is sufficient for me. Your mileage will be gauged by the religious institution and its man-made traditions which you favor over The Word of God.
Well then you have awfully low standards, because the "brothers of the Lord" doesn't settle the issue either way. Children from a previous marriage.