Not if they were all grown up and out of the house. Bear in mind, Joseph was in his 90's at the time.
"So there were no children from some fictional first wife."
Actually, there were six: four sons and two daughters. The wife's name was Salome.
Totally false.