What he said is that the song was about Caroline Kennedy. The lyrics to the song tell what he was thinking. I realize he was just trying to ingratiate himself in her presence when he said that, but to forget what his biggest hit song was about and say something that stupid is so Neil Diamond or schmuck'ish.
I won't argue with you there. No matter what you think about Neil, he doesn't deserve to be labeled a pedophile from a single ambiguous comment.