It seems to me that there is a factual basis to physical anthropology when they are dealing with fossils or DNA, but there is some guesswork because not everything can be explained--I don't think they have conclusively determined the relationships between all the pre-Homo sapiens hominid fossils, and at any time a new discovery can change what they think they know.
I consider that to be physiology, not so much anthropology.
It’s like a botanist knowing how plants relate to one another.
Outside of the physiology there’s not much of a factual basis, that I can see in anthropology, and a whole heap load of conjecture.
You can get that with History as well. Empirical history is only part of the overall discipline.