Paul calls James the Lord's brother.
There were also many women there, looking on from afar, who had followed Jesus from Galilee, ministering to him; among who were Mary Magdalene, and Mary the mother of James and Joseph, and the mother of the sons of Zebedee.
(Matt. 27:56; see also Mark 15:40).
James was a relative of Jesus - a more or less distant cousin.
It's important to realize that the Bible wasn't written in modern English.
'Brother' is often used as a translation of adelphos but adelphos does not mean 'brother'. A better translation - following the actual use of the word in scripture - would be 'kinsman' or perhaps 'close relative'.
Lot, for example, is called Abrahams "brother" even though he was actually Abrahams nephew.
Similarly, Jacob is called the "brother" of his uncle Laban.
From 1 Chronicles: Kish and Eleazar were the sons of Mahli. Kish had sons of his own, but Eleazar had no sons, only daughters, who married their "brethren" - the sons of Kish. These "brethren" were really their cousins.
Don’t you read passages from Paul that say “Brothers and Sisters?” Or “Brethren?”
Paul calls all followers brothers and sisters.