It wouldn’t matter to the ball if I was there to intercept photons that already contacted the ball.
We don’t know that. Because there’s no way to see what the ball would do when it is not observed. Meanwhile the photon experiment stands, observing does change the behavior of what is being observed.