Once one has a nice orderly system of differential equations governing the dynamics, the presence of a stochastic element in the genesis of biological diversity no longer makes a compelling counter to argument from design. Particle physics as presently understood has a stochastic element in the collapse of the wave function, but has such a beautiful, elegant feeling that no atheist brings it up as an anti-design polemic--it looks too much like the product of a brilliant mind.
My only real quarrel with Darwinism as science is the fact that its practitioners tend to formulate it in a non-falsifiable way (and I am a Popperian). My quarrel with certain Darwinists is that they are appealing to our ignorance of the non-stochastic part of what should be turned into a proper scientific theory (in the Popperian sense) when they draw philosophical inferences (the non-existence of God, the purposelessness of human existence, etc.) from the sketch of a scientific theory Darwin provided.