The Battle of Manzikert between the Byzantine Empire and the Seljuk Empire on 26 August 1071 was the point when the Byzantines lost Anatolia.
But the loss of EGYPT to the Arabs in 636 AD was the bullet that killed the Roman empire - it just took a long time to die.
When Gaul, Iberia and Italia were taken over by Germanic rulers, it didn't affect the economy of the Romans too much -- the bulk of the money was from the Greek east. When the Arabs took it over, it gutted 50% of the economy. An analogy would be if the entire lands that the USA won in the Spanish-Mexican war were lost to the USA - Texas, California, etc. gone
Egypt had been lost and recovered just a decade earlier (the Persians conquered Egypt briefly until Emperor Heraclius conquered it back (the reason for this was the stupid way in which Emperor Justinian II re-started a never-ending war with Sassanid Persia AND the way in which the Romans persecuted the Monophysite Copts/Syriacs/Armenians).
With the loss of Egypt, Rome became limited to Anatolia and the Balkans - halved in population and GDP.
With Manzikert, the loss of Anatolia meant the empire lost again 50% of its population and GDP.
The main causes were the weakness of the Byzantine state after death of Emperor Manuel in 1180. Series of coups and incompetent rulers led to separatism and fragmentation. Serbia and Bulgaria were already practically independent when the Third Crusade passed through. Also Armenians carved out a separate state. There was a rival Komnenian Emperor on Cyprus. This weakness was visible to Western Europeans (who importantly still seen it also as rich which is not a good combination with being weak).
By this time also the distrust and hostility between Latins and Romans had grown. Romans (Byzantines if one prefers) thought the westerners to be greedy barbarians with odd religious practices. On their part, westerners seen Byzantines as simply Greek (as opposed to Romans), cowards and ungrateful people. Westerners were also becoming more confident at the time.
Emperor Manuel also managed to come into conflict with Venetians, whose properties he confiscated in 1171. His successor Andronicus I massacred Pisans and Genoese in Constantinople.
This, the "Massacre of the Latins" pissed off the Venetians and they moved the Crusaders to support Alexios IV as claimant to imperial throne. They installed him in 1203 only to realise that the Empire, which lost a lot of its tax base due to separatism, could not pay what Alexios promised. He also did not have any effective army of his own (apart from still quite formidable Varangians).
Then in 1204 the Crusaders stormed the city.
This is the fragmented empire after the sack of 1204 -- Note that the Seljuk Sultanate of Rum was already huge
Chris Wickham speculates that:
“Had the events of 1203–1204 never occurred, it would be easy to imagine that a second Alexios I could have reunified the empire again and re-established its centrality as a European power which could have been more culturally integrated with those of the rest of Europe, possibly through the mediation of the Italian cities.(…) The Byzantine model might have become effective again, and maybe even more effective than it had been previously.”The Fourth Crusade did not necessarily cause the decline of the Byzantine Empire: but it certainly sped it up. The Empire had been in decline for the past quarter-century with the reformation of a Bulgarian state, the splintering of the Empire in Cyprus, the rapid distintegration of its ability to defend itself (exemplified first in the Norman sack of Thessalonica and then in the Latin conquest of Romania), all coupled with a series of internal issues. But it certainly administered the coup-de-grace to the Empire, after it which it could never really recover.
(Chris Wickham, “Medieval Europe” 2016).