Cousins or close family members (possibly raised by the Holy Family), but they are NEVER referred to as sons or daughters of Saint Joseph or the Blessed Virgin Mary. James calls himself a "servant" of Christ in his epistle, Jude states the same and says that he is the brother of James -- neither declare themselves to be the Lord's brothers.
I'm glad that you accept that the Lord built His Church on Peter's confession, giving Peter the Keys to the Kingdom of Heaven and continuing the Church based upon the Apostolic succession of the One Holy Catholic and Apostolic Church.
I read Mark 3, did you? Mark 3 clearly states that Jesus is asked about his physical brother and mother and Jesus responds what about them.
I believe the Church is built upon Jesus Christ. Emphasis Jesus Christ.