I take my cues from what I find in the Bible and how scenes from the Bible inform regarding concepts in modern Physics and Cosmology.
If you will, the where when of 1865 events is not only 'back', temporally, it is spatially in a very distant place, when you consider the motions out galaxy cluster, out galaxy, our solar system, out planet, and the background of the entire universe are compared moment by moment and location by location. Our motion is a revolving planet, an orbiting planet, a star orbiting in a rather remote arm of a galaxy which is moving with a cluster of galaxies, in some direction. Each of these motions have a different rate of location change, and due to gravitational fields effecting temporal reality, have different temporal shift rates.
Right. The coordinates ARE different.
An interesting (not scientifically rigorous, but interesting) take on this concept is Stephen King's novella, The Langoliers. In this story he explores the idea of "the past" being an actual PLACE.