You're arguing from consequences. Has the TOE been used as justification for certain atrocities? Of course. But that's not an argument against whether the TOE is correct.
Darwin showed that material causes are a sufficient explanation not only for physical phenomena, as Descartes and Newton had shown, but also for biological phenomena with all their seeming evidence of design and purpose. By coupling undirected, purposeless variation to the blind, uncaring process of natural selection, Darwin made theological or spiritual explanations of the life processes superfluous. Together with Marx's materialistic theory of history and society and Freud's attribution of human behavior to influences over which we have little control, Darwin's theory of evolution was a crucial plank in the platform of mechanism and materialism…
---Douglas Futuyma's Evolutionary Biology (1998, 3rd Ed., Sinauer Associates), p. 5