Jesus gave the authority to the apostles, none of whom were His brothers by a blood relationship. Nice try at the "gotcha" question.
It wasn’t a gotcha question. The James as the lead elder of the Church of Jerusalem was not the disciple and apostle James.