Churchill was an outcast in the Conservative Party at that time.
I think deep down most Conservatives thought Hitler was useful as a bulwark against Stalin. Churchill was the only one who saw, even as early as 1934, how evil Hitler really was.
—Churchill was an outcast in the Conservative Party at that time.—
The origins of his outcast state within the Tories had nothing to do with Hitler, however, and stretched back all the way to the 1900s.
As for bulwarks, Churchill also initially thought the Japanese invasion of Manchuria was justified, somewhat, due to the Soviet menace next door.