You do make a good point in that, from our perspective and viewpoint, the past (like this morning's great cup of coffee) is behind us and only exists in our memories unless memorialized in writing somewhere).
If we could get outside of our constraints (but who can?), we might be able to see all of eternity - past, present and future as well. That is only available to a being who is outside of our constraints.
In my worldview, that escape from time & place only exists in God, who created everything, including time & space. And I can't wrap my mind around any of that, so I live by faith.