The Nephilim perished in the great flood. And remember, the Bible says that Noah was perfect in his generations.
There were other giant people that descended from Noah, most notably the descendants of Anak, and their relatives the Rephaim. The Bible does not call Goliath and his unnamed brother anything except Philistines.
The Nephilim were on the earth in those days—and also afterward—when the sons of God went to the daughters of humans and had children by them.