Yes, every atom of your body existed a billion years ago, just somewhere else. If you travel back or forward in time, you would essentially be ‘creating’ mass that did not exist previously in that universe, which is a no-no..............
Everything is going forward in time though.
I do agree with you that the "replicator" aspect is forbidden. Still made for a good story.