Years ago I studied this topic beginning with the scriptures only. I came pretty much to the same conclusion you just posted. The Deuteronomy 32 matter came later as I had trouble understanding how the Masoretic text could have this error.
But when I looked into early church writings, commentaries, and non-canon texts like the Book of Enoch, I found that this view is commonly held.
I grew up hearing that demons were fallen angels, but when I studied for myself I found this distinction in scripture between fallen angels and the Nephilim.
Apparently there was a very small number of angels who were involved in this particular rebellion. And I don't think it is unreasonable to think this event may be repeated as a fulfillment of Christ's prophecy about the end times being like “the days of Noah”.
The way different theologians and ministers treat things related to Gen 6 is shocking when, after digging into it, it becomes quite clear what the Bible is saying.
I have explained this to Christians who, with mouths agape, couldn't get their minds around it. Even Paul drives the point home more generally in Ephesians 2:2 and 6:12:
Eph 2:2
Wherein in time past ye walked according to the course of this world, according to the prince of the power of the air, the spirit that now worketh in the children of disobedience...Ephesians 6:12:
For we wrestle not against flesh and blood, but against principalities, against powers, against the rulers of the darkness of this world, against spiritual wickedness in high places.I have shown people, for example, that the very important phrase "sons of God" (Bənē hāʾĔlōhīm) is only used in the OT to refer to angels/spiritual beings and they still insist it must mean something else. As more than one Biblical commentator has said, "You are in a spiritual battle. If you refuse to face that fact, you probably aren't winning the battle."
In this vein, let me suggest Heiser's Unseen Realm. You have probably read it, but if you haven't you have a real treat ahead.