According to Wikipedia:
"In the Hadrianic war 580,000 Jews were slain, according to Dion Cassius (lxix. 14). According to Theodor Mommsen, in the first century C.E. there were no less than 1,000,000 Jews in Egypt, in a total of 8,000,000 inhabitants; of these 200,000 lived in Alexandria, whose total population was 500,000. Adolf Harnack (Ausbreitung des Christentums, Leipzig, 1902) reckons that there were 1,000,000 Jews in Syria at the time of Nero, and 700,000 in Palestine, and he allows for an additional 1,500,000 in other places, thus estimating that there were in the first century 4,200,000 Jews in the world. Jacobs remarks that this estimate is probably excessive."
http://en.wikipedia.org/wiki/Historical_Jewish_population_comparisons#Ancient_and_medieval_times
Notice there is NO MENTION of any large population of Jews in Babylon and certainly not the "literally millions" you speak of. Church fathers have always held that Babylon was code for Rome and there is no account in Acts or Paul's letters of anyone going to Babylon, in fact the only references are Peter's and John's in Revelation. Now, Revelation speaks EXTENSIVELY of Babylon and it aludes to their power, yet we KNOW THERE WAS NO POWERFUL BABYLONIAN EMPIRE IN THE FIRST CENTURY. Therefore, we must conclude that Babylon means Rome.
At the beginning of the present era there were many conversions to Judaism all over the Middle East. In about 40 CE, in northern Iraq, the Royal Family and many of the people of Adiabene became Jews. It is estimated that there may have been as many as one million Jews around Babylonia at that time.
The above paragraph is about one third of the way down here.
The Church Fathers were obviously wrong here also as even Josephus says "The Ten Tribes beyond the Euphrates were of such a large number no one could estimate the total". [Antiquities XI, Chapter X, Paragraph 2]
Who do we believe....your guy or mine. The point remains....Our Lord told the Twelve to "Go to the Lost Sheep of the House of Israel". He didn't mean Rome.