Nothing in Scripture calls those "brothers" the son of Mary, or of Joseph for that matter. James and Joses are called the "brothers of the Lord" but they are also called "sons of Alphaeus".
As the Lutheran scholar Joachim Jeremias has noted, an obedient Jew can't absolve another Jew of his duty under the Law, so Jesus giving Mary over to John (not a blood relative) to be cared for was committing a serious sin -- as practically his last act on earth, no less! -- if he had living siblings.
The Greek word "adelphos" is used in Mark 1:16 and 1:19 to describe the pairs of brothers selected as Jesus' disciples.
In Mark 6, when Jesus went to "his own country" (i.e., where he grew up) and spoke to his own townspeople, they remarked in Mark 6:3 "Is not this the carpenter, the son of Mary, the brother of James, and Joses, and of Juda, and Simon? and are not his sisters here with us?"
That same Greek word adelphos is used here... and "adelphē" is used to describe his sisters, for that matter.
Of all people, this group knew Jesus had brothers and sisters... they where certainly not talking about 'spiritual' relatives... it was obviously actual relatives, and the context makes it crystal clear. I don't know how you can possibly read it any other way... Catholic sensibilities notwithstanding.
Paul identifies James as the Lord's brother. The Greek is clear on this.