True that you can’t personally go back in time but no physical evidence in geology, physics, chemistry, etc. contradicts this assumption and, it is support by overwhelming observations.
“no physical evidence in geology, physics, chemistry, etc. contradicts this assumption”
Lack of contradiction of an assumption isn’t the same as experimental confirmation.
“it is support by overwhelming observations”
If your assumption is correct, we’ve only been able to observe a statistically insignificant portion of the potential data, so that statement is meaningless.