Truly I believe they thought there was an air of the exotic about the Muslims. The French view themselves as sophisticated and the air of mystery somehow added to their nuanced cultural palate in a way that crass Americanisms could not.
And let’s not forget that there is still an anti-Jew undercurrent in France. They would never, ever admit it. It’s there nonetheless.
I think you hit the nail on the head.
The muslim influx to France is a result of their former colonialism. No “mystique” about it.