I've thought of the same thing. Even the rotation of the earth must come into play. For example, if I could time travel to 12 hours ago, my physical location might well be somewhere in Asia (not even considering the variables you just mentioned). So any "time travel machine" would have to also make the necessary corrections for physical location - not only in the universe but for movements in the galaxy, solar system and even the rotation of the earth (and the shifting of continental shelves).
The more you think about time travel, the more difficult and complex it becomes.