From the NAB
The brother of James . . . Simon: in Semitic usage, the terms "brother," "sister" are applied not only to children of the same parents, but to nephews, nieces, cousins, half-brothers, and half-sisters; cf Genesis 14:16; 29:15; Lev 10:4. While one cannot suppose that the meaning of a Greek word should be sought in the first place from Semitic usage, the Septuagint often translates the Hebrew ah by the Greek word adelphos, "brother," as in the cited passages, a fact that may argue for a similar breadth of meaning in some New Testament passages. For instance, there is no doubt that in v 17, "brother" is used of Philip, who was actually the half-brother of Herod Antipas. On the other hand, Mark may have understood the terms literally; see also Mark 3:31-32; Matthew 12:46; 13:55-56; Luke 8:19; John 7:3, 5. The question of meaning here would not have arisen but for the faith of the church in Mary's perpetual virginity.
Oh! A footnote. Nevermind.
Jesus is not the “son” of Mary because He is the “son” of Joseph and we know that isn’t so. John is also the “son” of Mary so we can’t be sure if the writers are just using “son” metaphorically.
John is not the “brother” of James and Peter is not the “brother” of Andrew because we know that “brother” is used universally to mean friends and near relatives.
Father of course can mean many things like “father” of lights, we aren’t to call any man “father” here, there’s our “father David”, “father” Abraham, “father” Jacob, “father” Isaac, your “father” the Devil, “father” of many nations.
Based on your logic we can’t be sure of any relationships in the scripture since they are used for blood relatives, close relatives, far relatives, close associates, and concepts.
So please explain, based on your exegesis, how you arrived at Mary being the genealogical mother of Jesus?