Where are you imagining the difficulty? There is no anchor holding you against drift. Randomness does not force a net zero change. In fact, random drift will move you in proportion to the degrees of freedom and the square root of the time factor. IOW, yes. You are wrong.
You stay unchanged when you're at a local fitness maximum. There are active conservative pressures to not change when you have it about right.