Scripture tells us about James' parentage: he's the "son of Alphaeus" (Mt 10:3), not Joseph. Some people have invented the notion that "James, the brother of the Lord" and James, son of Alphaeus, were two different people. Neither Scripture nor tradition support this. We know of two apostles named "James," one the son of Zebedee and the other the son of Alphaeus, not three.
Nobody except Jesus is ever described as a son or daughter of Mary by blood in Scripture. If we're going to "be silent where Scripture is silent," we have to be silent there.
It's also trivial to prove from Scripture that Mary had no blood children, other than Jesus, alive by the time of the Crucifixion. On the Cross, Jesus gave Mary to John, who everyone agrees was not a blood relative of either of them. If Jesus had living brothers or sisters, he sinned gravely in almost his last moment of earthly life by absolving them of one of the most important commandments, "Honor thy father and thy mother."
“Nobody except Jesus is ever described as a son or daughter of Mary by blood in Scripture. If we’re going to “be silent where Scripture is silent,” we have to be silent there.”
You choose a silent point to prove a silent point? Yet, Scripture isn’t required to meet your criteria of proof.
Since there are people identified as brothers, and Greek has a word for cousin and a word for brother, and the word for cousin isn’t used, your burden to make it into something else is heavy.
If you want an argument from silence, focus on the notion that Mary stayed a virgin - in violation of Paul’s commandments to couples to only withhold sex for a short time.
These brothers [of the Lord] have been regarded:
(a) by the Orthodox churches as sons of Joseph by a previous marriage (the Epiphanian view)(b) in Roman Catholic interpretation as Jesus first cousins, the sons of Mary wife of Clopas, who was the Virgins sister (Jn. 19:25; the Hieronymian view)
(c) by Protestant exegetes as Jesus uterine brothers, sons of Joseph and Mary (the Helvidian view). This last view accords best with the natural implications of Mk. 6:3, where the context suggests that the brothers, together with the sisters unspecified by name, were, like Jesus himself, children of Mary.
Constable, T. (2003). Tom Constables Expository Notes on the Bible