The Jews -- especially the orthodox -- also are isolationists and nonconformists. Their self-imposed "otherness" makes them a natural target for suspicion. The "other" is always easier to resent.
The very fact that one wouldn't bash the Amish for being "isolationist" that makes them "suspicious" reveals that anti-Jewish hatred is blind. Always covered with a mask/excuse. I mean, how dare someone Jews want to preserve their tradition...?