I don't understand why Catholics continuously throw that out there when Jesus already told anyone who is willing to read the scriptures why he didn't call them back...And it had absolutely nothing to do with whether what he said was literal or a metaphor...
The very fact that Jesus was telling them to do something so contrary to the clear teaching of Scripture was enough to tell them that it was a metaphor.