James also said in the same discussion:
James 2:
23 And the scripture was fulfilled which saith, Abraham believed God, and it was imputed unto him for righteousness: and he was called the Friend of God.
So there is that right in the middle of James point.
The misunderstanding comes when people pluck verses from context.