I don't think so. We've been watching for a lot of years worth of seconds already and there's been no detectable outward drift. If, say, the Earth were 6,000 years old now and there has been over this time an accumulating acceleration, the effect might have become at least measurable with the sensitive technology we've been using since the dawn of the space age.
That's your model. The effect is tiny, but it builds and builds until it's whooshing through the galaxy and we're flying right out of the solar system. If we're supposed to be flying trillions of miles out after a billion years, we should be able to tell we're moving after 6,000.
Anyway, the model is still bad. One of your wave-aways was this: "The other noise you were talking about is in random directions and therefore cancels out."
Just in one case, I don't think the meteors do cancel out. If nothing else, I think there's a bias in favor of impacts in the direction we move. It's similar to how there's more rain on your car's front window than the back when you're moving. That's a decelerating effect which would tend to more than cancel your teeny tiny solar sail push.
Thus, maybe your winning ICR contest entry should be titled, "The Earth Would Have Fallen Into the Sun by Now if it is Old." However, orbital mechanics are based upon fairly stable equilibria and are not easily degraded to the point of catastrophe except when big things start banging into each other.
But your effect is so teeny tiny that in any event the Earth just eats it with the mushiness of its atmosphere, oceans, snowpack, etc. Earth is not a SuperballTM, it's a bean bag.