And yet Paul had to correct Peter on his error of not eating with the Gentiles...Hmmmmmm.
You should look up the difference between infallibility and impeccability. You should also see with what authority the gathered apostles and presbyters in Jerusalem invoked when they wrote to the church in Antioch:
It is the decision of the holy Spirit and of us (Acts 15:28)N.B. that this was not just the decision of the apostles but of the apostles and presbyters. The original apostles had already gathered other men to themselves to share in the governance of the church and invested with the authority of the Holy Spirit. To accept Sacred Scripture one must also accept this abiding authority which Scripture proclaims.
James had to correct the vain man who thought one could be saved by Sola Fide. Paul's reproof was behavioral, while James' was doctrinal, upon which millions have foundered.