Up until the days of Josephus, the whereabouts were clearly known.
I used to go round and round with one professor in bible college about this. His view was that the sons of Seth marrying the daughters of Cain caused the Nephilim, which he claimed were simply "tyrants". Because the Hebrew meaning can also be translated as "tyrants". That is another whole discussion of its own on how you get tyrants from marrying a non believer... But that was his position and later I learned had been the position of many churches. I was shocked. At the time, as a new believer, I was dumbfounded at how brain washed/dead (take your pick) many professors were at a place that trains pastors and missionaries and teachers.
I hadn't heard that view before until, of all places, I was in a Christian College! It was such a much further stretch to believe that than the actual truth of the matter.
Book 5 Ch 2 sec 3. From Antiquities of the Jews (Roman Historian Flavius Josephus has numerous references to the Giants. Here is one:
3. For which reason they removed their camp to Hebron; and when they had taken it, they slew all the inhabitants. There were till then left the race of giants, who had bodies so large, and countenances so entirely different from other men, that they were surprising to the sight, and terrible to the hearing. The bones of these men are still shown to this very day, unlike to any credible relations of other men.
ENDNOTES (from Book 1 Ch 3)
(11) This notion, that the fallen angels were, in some sense, the fathers of the old giants, was the constant opinion of antiquity.
text of Book 1 Chapter 3:
For many angels (11) of God accompanied with women, and begat sons that proved unjust, and despisers of all that was good, on account of the confidence they had in their own strength; for the tradition is, that these men did what resembled the acts of those whom the Grecians call giants. But Noah was very uneasy at what they did; and being displeased at their conduct, persuaded them to change their dispositions and their acts for the better: but seeing they did not yield to him, but were slaves to their wicked pleasures, he was afraid they would kill him, together with his wife and children, and those they had married; so he departed out of that land."
Note: there are at least 6 places in Antiquities where Josephus mentions the giants. (Books 1, 3, 5, and 7)
***
This notion, that the fallen angels were, in some sense, the fathers of the old giants, was the constant opinion of antiquity***
They were set on changing God's creation. They were all of one language, of one accord. We learn from the Day of Pentecost the power of 'being of one accord.' The Lord thought to change all of that with a flood; but after the flood there continued a move to change the order of God's creation. (Today we would term it 'genetics'.) Their hearts were forever set on doing evil.
God had promised that there would not be another flood, so he set his sign - a rainbow - in the heavens. Therefore the Lord God destroyed the tower (Babel) built by giants and scrambled all the languages of the post diluvian world.
This is the last time that God has directly interfered in world politics. The scrambling of tongues made understanding across cultures difficult to impossible; the fallen angels could understand the various tongues but could only sew distrust when trying to communicate various universal ideas to humans in such confusion.
Today the enemy is working prodigiously to open a portal back to pre-Babel times. For example, they have managed to get erected a 'gold' statue of a female goddess next to Moses over the entrance of a NYC courthouse:
this leading up to the arrest of Donald Trump.
Spiritual warfare has become a central part of our age... mene mene tekel upharsin...