No, I would not; but that does not seem to be what was happening in Noah's times.
Look up what "nephil" means in the Bible-time Hebrew. There is nothing that says those called "nephilim" (= plural of nephil) were physically unusual in any way. Strong's Concordance is not helpful in this, because all that you see there is how a word in the original language was translated into English by the translators of the Crown-Authorized version, and they were not always right.
In this particular case, They followed the translators of what we think is either the Septuagint--which translation of the Old Testament has its own lack of rendering the Hebrew and Aramaic (Chaldee) properly into Koine Greek--or the Vulgate, in which Jerome (though mainly translating from the Hebrew)--followed the Septuagint in Genesis 6:4 and not the Hebrew.
Recognizing that only Moses who wrote the word "nephelim" (and that only three times) could tell us exactly what he meant by that term, we must go to other Hebrew literature to discover what it meant to his culture. And that is what lexicographers do.
But firstly, all of the Nephelim of Numbers 13:33, though tall, were necessarily humans, being descendants of the eight humans borne through the Flood, on the ark. Furthermore, these totally human Nephilim were also "mighty ones," warriors, likely because of their size. That does not necessarily mean that all nephelim were warriors, or that they all were tall, does it? In fact, we only see an exaggeration of their size by the returned disbelievers, not by Caleb or Joshua who were more realistic.
Secondly, Young's analytical Concordance does not accord the nephelim of Genesis 6:4 as being large, only mighty by their coordinate descriptor "gibbor" the Hebrew term mighty and/or strong; and "fallen ones" as the definition of nephilim.
Thirdly, the highly-regarded Hebrew-to-English lexicon (click here) of Dr. William Genesius, professor at Halle, gives (in my edition) about 24 column-inches of fine print on what "Nephil" means on pages 557-558, and none of it mentions anything about giants.
Now, you may wish that Genesis 6:4 refers to "nephelim" as "giants," but in that we will part company, with my foundation being correct translation of the Hebrew text as the issue, and correct interpretation/exegesis as the result.
For me, nephilim are humans who have fallen from fellowship with God by rejecting the faith of Seth, Enoch and Noah. They are men who are not sons of God. And the context does not say that the faithful daughters of God were not fair to look upon and graceful. What one might suspect is that they were not quite as easily available for premarital recreational sex as the daughters of fallen humans, especially for local famous heroes. Considering this root cause, even more phony religionists, hedonists, and God-deniers overwhelmed the faithful, even to extinguishing the gospel of grace, such as is happening now.
God is going to wipe them out once more, take away Satan's influence, and for a thousand years see rebelliousness occurring even under direct and close supervision that has immediate consequences.
Only in the nephelim thing, not in general. In most everything I agree with and encourage you.
I should have made this a little clearer.