The KJV pulls a little shell game, here. The name translated as “Joses” is the same name translated elsewhere in Matthew as “Joseph,” and the name translated as “Judas” is the same as the name translated elsewhere as “Jude.”
So, the question the Jews asked was “And his brethren, James, and Joseph and Simon and Jude?”
This brings to light two problems with interpreting “brethren” narrowly:
Jesus was certainly the firstborn. Why would a second (or likely third, since James was apparently also older) child be named after the father?
James and Jude are the names of his two apostles who are identified as sons of (or from) Alphaeus. Their mother is also called “Mary.” But this Mary is not the same Mary as was the mother of Jesus, for she is listed separately with Mary, the mother of Jesus.
The ancient Jews did not distinguish between “cousin” and “brother.” Someone translating from Hebrew to Greek might well use “adelphos” (brother) instead of “anepsios” (cousin), especially since “anepsios” also means “nephew,” and the Greeks did commonly use “adelphos” to refer to brothers, not cousins. In fact, “anepsos” is where we get the word “nephew,” and King James translated it as “sister’s son” in Col 4:10, the only place where it occurs.
Jesus was not the son of Joseph, but of God, and He was named by God.
Matthew 1:23 "Behold, the virgin shall be with child and bear a son, and they shall name him Emmanuel," which means "God is with us."
Further, if James was older than Jesus he would have accompanied his parents to the Passover.
Luke 2:
41 Now his parents went to Jerusalem every year at the feast of the Passover.
42 And when he was twelve years old, they went up according to custom;
That is, assuming James was not so much older he was no longer with the immediate family. This leaves the whole nuther problem of the "much older James and his brothers accompanying Mary during Jesus' ministry.
(Of course this is assuming Joseph ever had any children and was also, as according to Jerome, a Perpetual Virgin.