When I realized he was a total jerk was when he got angry at David Prowse for revealing that he got to show his face in Return of the Jedi and went back and reshot the scene with a different actor, so Prowse wouldn't have his face on screen. It was so much effort to be so petty.
My time I realized he was scum was when he bragged about the Ewoks being the Vietcong and the Empire was America. Made even worse when development notes indicated the Rebels shared the same inspiration as the Ewoks. Maybe also his having Obi-Wan give a nihilistic remark to Anakin in Revenge of the Sith.