I beg to differ. Your existence and mass in the 4 dimensions are set (I believe we’re in agreement on this part). If you go “back” to an earlier period in time you will not exist in the current period. Matter and energy are going to be conserved in there somehow.
Now, you may have to be turned to energy and “re-written” on that past period, but you are going to disappear in the here and now going forward (barring consideration of “many universes” theory) and the net matter-energy budget for the 4 dimensions is going to remain balanced.
The question that would arise to me out of this is that you would alter the net energy balance of the universe. I.e. as a whole, the amount would remain the same, but you would be changing the concentration of it back on the time pathway at some point. How this would re-equilibrate and what that process would mean/do to the past and present would definitely warrant study before doing it.
The total mass and energy of a particular universe must always be in balance, and will be one way or another.
The only way I can see of possibly being able to time travel would be to extract an equal amount of mass/energy from the time to be visited and transferred to the time being left, minus whatever energy was used in the transference of the two masses.
Every atom and sub atomic particle would have to be accounted for, because if they weren’t, an imbalance would occur that would be immediately corrected in a very violent manner.
Perhaps small imbalances of a few atoms of air or so would be acceptable as long as the energy of their destruction was nullified by a special suit or vehicle capable of withstanding the energy of dissolution........