The thing I find interesting is how the bible describes Nimrod as "a mighty one on the earth". This kind of phraseology typically is used when describing the offspring of the fallen angels (the sons of God saw the daughters of men, and took whomever they chose to be their wives... and those offspring were called mighty men of renown"... Nimrod could have been a halfbreed, part human and part fallen angel... could be...
I suggest they never went away...they just hid. Just a suggestion...no proof.