Just how would visiting parallel dimensions “visiting the past?” If you are going to the past you have to go within your own “dimension” or it is not the past.
Perhaps because they would have had a common point in time prior to splitting off on their own separate paths?