Without the Electoral College, New York City would be able to use massive vote fraud to increase its influence over the election. With the Electoral College, once a candidate has a comfortable margin in a particular state (perhaps 5%+) there is no incentive to increase that margin by pandering to the extremes within that state.
Madison would agree with you. Guarding against demagogues was one of the reasons the Electoral College was proposed at the Convention.