Yes, you have a point. And another point is that fitness and survival involves a trend toward the opposite.
Moving back to the theme of intoxication for a moment, a drunkard's walk from a wall will trend away from the wall. Stochastic change from a lower limit of complexity (bacteria) will produce some instances of greater complexity, but the modal trend remains near the lower limit of complexity.
And bacteria remain the dominant life form on earth, by any measure relevant to survival.