Mark 6:3
"Where did this man get these things?" they asked. "What's this wisdom that has been given him, that he even does miracles! Isn't this the carpenter? Isn't this Mary's son and the brother of James, Joseph, Judas and Simon? Aren't his sisters here with us?" And they took offense at him.
Galatians 1:19
I saw none of the other apostles—only James, the Lord's brother.
Acts 1:12-14
12Then they returned to Jerusalem from the hill called the Mount of Olives, a Sabbath day's walk[a] from the city. 13When they arrived, they went upstairs to the room where they were staying. Those present were Peter, John, James and Andrew; Philip and Thomas, Bartholomew and Matthew; James son of Alphaeus and Simon the Zealot, and Judas son of James. 14They all joined together constantly in prayer, along with the women and Mary the mother of Jesus, and with his brothers.
That's all well and good, but Jews of the time were identified as "son of" not brother of. There is no doubt in my mind that these others were close to the Lord growing up, it is very possible that they were raised by the Holy Family and were referred to as brothers. But the fact still remains that they are not identified as the children of Mary because that is untrue. If they were, Jesus would not have had the option of leaving His Mother in the care of John.