You can remove loose impediments til the cows come home (unless you’re in a hazard). The problem is when your ball moves because of YOUR actions.
Every player on that tour knows when their ball moves. Many have cost themselves tournament titles by calling it on themselves when no one else ever could have known, or seen the infraction.
That goes for everyone. I definitely know when mine move.