That explanation might fly in Sunday school, but scholars know the truth to be more complex. If you look in Middle English literature, for example, you will only rarely find the Muslims posited as "threatening." They were seen as strange and exotic heathens, yes, but not as military threats.
Remember that the most horrific crime in the Canterbury Tales (as seen in "The Prioress's Tale") is committed by the Jews, not the Muslims.