Much more likely that they were a cousin species of humans - like Neanderthal.
I doubt very much than any of this DNA would be of the sort that would make a man a “giant” or “of much renown”; doubtful that it would have any effect on phenotype at all.
What about Goliath?
If they were a larger race, they may well have made giants. I think there have been dozens of hominid species. Our recorded history goes back about say, less than ten thousand years. Our earliest traces are about 40,000 years old. Who the heck knows what happened earlier than that? Nobody. The trouble with fossils is that there are so darn few of them - just the fact that it got fossilized makes it an outlier. Is it possible that the out of africa theory is the first known example of white flight?
Whatever the case, all of our DNA is close enough that we are RELATED we all came from a small tribe, a remnant. Cousins.