It's the phiz. That noisome, supercilious smirk is the default setting for his face. That is the obvious off putting thing.
Another and more subtle thing is his eyes. Even when talking or emoting, there is the disconnect with his eyes. They are always scanning and assessing his audience reaction. Always calculating, always disengaged except for that assessment. That is the habit of a con, a hard sell artist, a litigator. It is the look of a cynical manipulator. Clinton was much better at concealing that when he was reading his marks. (FReeper Psalm 144)