He who is not with me is against me.
For he that is not against us is for us.
At once this may seem to present a paradox. But for sure, it does not. Jesus knew well the extremisms that existed in his days. The Pharisees and Saducees who were extremist rigorists and the lapsed Jews who were comparable to our modernists. Jesus was basically telling his disciples that extremism was not of the Gospel.
I'm not sure what saint said it (Aquinas maybe), that extremism was a vice that corrupted the virtues of prudence, patience, humility and love!
I think that what you are thinking of is the Aristotelian/Scholastic definition of "vice" as "a deficiency or an excess of a virtue" -- e.g., the virtue of courage is opposed by the vices of cowardice on the one hand, and being rash on the other.
Not exactly the same as a accusing ideological "extremists" of vice.