It matters very much.
In keeping silent about evil, in burying it so deep within us that no sign of it appears on the surface, we are implanting it, and it will rise up a thousand fold in the future. When we neither punish nor reproach evildoers, we are not simply protecting their trivial old age, we are thereby ripping the foundations of justice from beneath new generations.
Aleksandr I. Solzhenitsyn, The Gulag Archipelago 1918-1956
Langston Hughes had this to say about that:
Looks like what drives me crazy
Don't have no effect on you--
But I'm gonna keep on at it
Till it drives you crazy, too.