You have the testimony of Ireneaus and Tertullian. Absent evidence to the contrary, that should be enough. Besides, this is as unexceptional as asserting that a Frenchman might go to live in Paris. Rome was a magnet drawing people from all over the world and had a large Jewish population.
Neither lived contemporaneous with Peter. Tertuillian AD 145-220, Ireneaus AD 120-202. So anything they have to report would be hearsay. They never claim to have actually seen Peter walking the streets of Rome.