Granted but let’s confess to a fair degree of mutual ignorance about the intricacies of travel through space/time. The concern you have may be addressable by the eventual technology. The scientists and engineers of 100,000 years hence are not bound by our lack of imagination or ignorance. This conjecture is so far out of the box that there may as well be no box. I don’t know how old you are but technologies unimaginable in my youth are already obsolete.
For a perspective on interstellar travel, imagine for a moment that you could view the galaxy from a right angle to the rough disc of our spiral galaxy. Where is Sirius in relation to Sol and where will Sirius be ten years from 'now' in relation to Sol? Now ask yourself how people six thousand years ago understood the concept of precession. Add to those pieces of data that a photon cross the universe in a null-path, not leaving the present of when it was emitted.
Human conceptualization of what dimension Time is is, well, very primitive still. we have apretty good grasp on waht it means to move through spacetime, but we as yet have not conceptualize dimension Time in a like fashion to dimension space (the linear, planar, volumetric conceptualization of dimension Space). And yet we have lots of clues that dimension Time has variable expressions! General relativity gave us a huge clue that dimension Time is not some background static through which we travel.
What I'm aiming at is the notion that just as we think in terms of moving from point A to point B as a linear motion "in" dimension space, we need to grasp a better notion of what it means to be 'here' and 'there' as a linear 'now' which is but one expression of a more complex planar and volumetric dimension.
We actually have clues regarding this notion, with the Christophanies, with the disappearance of Jesus from the tomb, the sudden appearnaqce of Jesus in the locked and shuttered upper room, and even in the fifth chapter of Daniel.
The foruteenth chapter of John's Gospel has a very interesting exchange between Jesus and Philip. Jesus avoided explaining a 'tesseract' yet he described His realtionship to The Father using the conceptualization as best Philip might comprehend it.