There is something else that I did not mention previously, mainly because I do not think it was a major reason for universal hatred, but may have played an important role in specific expulsions, and that is at certain points the non-Jews owed them money that they had loaned them and getting rid of the Jews was an easy way to get rid of their debts.
I've also read that borrowing actually went agains church teaching during the Roman Empire. So it was considered immoral, yet out of necessity I'm sure many people utilized the Jews to borrow money.