As pointed out, there are no extant instructions from him on the subject. St. James certainly would have known that both leavened and unleavened bread are just that, bread, and so both are valid matter. Implicitly, therefore, his lack of instructions to use only leavened bread shows that it doesn't matter.
"his [St. James'] lack of instructions" assumes facts not in evidence.
You are trying to make a case from nothing. According to the Tradition of the Church, azymes are not permitted. We have a positive statement to that efect in one of the canons of the Council in Trullo. And then on top of that we have the reference in Canon 32 to the Liturgy of St. James and how it came into being. What you have is your unswerving devotion to wafer worship. That is why we call you folks azymites.