He is right : he wasn't there--not even in the right century--and the writer of Matthew was. But he is certainly right about one other thing, that the magi had marvelous knowledge of the movements of the stars and planets. We also know, or should know, that we have forgotten most of the things that have happened since man appeared on this planet. Some things we have not.
Here are some "wise men:"
Daniel 2:48 Then the king made Daniel a great man, and gave him many great gifts, and made him ruler over the whole province of Babylon, and chief of the governors over all THE WISE MEN OF BABYLON.
In 586 B.C. the Babylonians invaded and captured Jerusalem. Thousands of Jews were forced to relocate and live in Babylon. Though many returned to Jerusalem at the end of this "Babylonian captivity," the area continued to have a large Jewish population for many centuries. This made it likely that wise men of Babylon knew of and were interested in the prophecies of a coming Jewish Messiah.
The 70 week of Daniel 9 epecially, it gave the general time for the wise men to look for the messiah, 483 years from the Cyrus decree “unto the messiah.” The wise men would have known that they were in the general time frame when this would occur. The time was nigh.
In the Old Testament Book of Numbers, the prophet Balaam says, "I see him, but not now; I behold him, but not near-a star shall come forth out of Jacob and a scepter shall rise out of Israel..." The wise men would have known of this prophecy. They would have connected the birth of the Messiah with a celestial sign.
By the end of the first century B.C. the wise men of Babylon would have been anticipating this sign for hundreds of years. Whatever the sign was it would have to be unique and in the time frame of the Daniel 9 prophecy.