I wouldn't say Muslims "drove" the system, though they were heavily involved.
But so were Sephardic Jews, Spaniards, and English -- not to mention Africans themselves. And other groups.
All true, with the Spaniards holding Muslims and Jews as accountable with the Inquisition, with the latter holding Western Christianity accountable for the inverse.
History is replete with causes for grievance.