There’s a lot of guessin goin on
That's going to be the case in any historical (and therefore, not directly observable) science-- archaeology, paleontology, forensics. But with techniques like DNA analysis, the guesses are becoming much more educated.