"And knew her not till she had brought forth her firstborn son: and he called his name JESUS" (Mt. 1:25 AV)
A very good indication that Joseph "knew" Mary after Jesus arrived, and she was also thereby no longer a virgin, but she and Joseph participated in God's commandment to be fruitful and multiply, rather than live unnaturally as mutual celibates, thereby defeating God's commission for them. Nowhere does Scripture record that they were not to enjoy the bliss of normal and complete family interactions.
Furthermore, to respond to the Luke 2:41-51 passage, there is no indicatiion there that other children of Joseph and Mary were not with them. In fact, their attention might well have been distracted by needful attention to other siblings, two or more years younger than Jesus.
FWIW
There is nothing said about intimate relations between Joseph and Mary after Jesus was born.
“knew her not” meant no intercourse had existed between them.
“not till she had brought her firstborn son and he called his name Jesus” meant she had no children previously and the first child was Jesus.
Anything after is pure speculation.
The Orthodox codified the terms ‘brother and sister’ to mean siblings and cousins in order to stop the speculation which existed in generations immediately following the crucifixion. It was important to early church leaders to keep the laity focused on the message of Christ and not the speculative fringes.
As Zefferelli had minor actors argue and scream during the film (my favorite) shooting of the trial of Jesus before the Sanhedrin “ I heard him say he would build the Temple in THREE days!” and another chiming in “No, I heard him say in TWO days!”. This is a sad part of human nature to zero in on trivia and not see the true important message.
Same with James as sibling brother of Jesus.