When you say that she didn't give birth to James, you have no proof of this. You are telling others that they can't make statements that James was the biological brother yet you assert that Mary positively did not give birth to James.
You can't have it both ways. If there's not enough evidence for one viewpoint then there's not enough for the other, either. And saying it was all decided by 500 A.D. doesn't wash, either. If it were settled we wouldn't be discussing it here.