You wrote: "Babylon existed then."
Are you so sure it was worth the effort at that time?
"How do we know Peter wasn't there at the real Babylon?"
We have no reason to doubt the sources. Even Christians in Mesopotamia believe Peter went to Rome. Besides, which Babylon? There were two you know? The second one was in Egypt. Did you even know that?
But we are talking about Rome, Italy not Babylon, Egypt.