I think the verse in Genesis 6 should be split in two, with no connection between Nephilim and giants.
If you consider an earthly interpretation, it was men of God fancying daughters of atheists and ended up marrying them creating kids with no foundation. Giants were powerful men of their day and they were tyrants.
Just like today.
And of course, the backstory provided by the extra-biblical books, like 1 Enoch and Jubilees, adds further detail—describing the Watchers, their descent to Earth, their union with human women, and the birth of the Nephilim. These texts expand on Genesis 6 and reflect how it was understood in ancient Jewish world.
As an aside, fragments of both 1 Enoch and Jubilees were found among the Dead Sea Scrolls, showing that these interpretations were taken seriously by ancient Jewish communities well before the New Testament era.