But you're right. Morality isn't solely defined by Christianity. Let's just say that at its nadir, Rome was capable of the depravities described in this article, and that that level of amorality, accompanied by the subsequent decline in social and military unity, contributed substantially to the fall of the Empire. By the time Constantine (and even Diocletian) managed to recover some sense of integrity, there was little left of Glorious Rome.
After Constantine, only one emperor, Julian the Apostate (360-363), was not a Christian. Julian made some vague rumblings about disfavoring Christians, but these were perfunctory, and Julian's reign was rather short in any case. These don't really amount to "attempts" to return to paganism.
By the time Constantine (and even Diocletian) managed to recover some sense of integrity, there was little left of Glorious Rome.
Again, you are too broad. Diocletian's reforms crippled the Roman economy, which had already been weakened by a century of almost continual civil war. But Constantine's reign came at the beginning of the fourth century, and at this point in time Rome was still incomparably mighty and in possession of much of her past glory.
Some historians, most notably Gibbon, have argued that Christianity actually hastened or even caused the fall of the Roman Empire, through a variety of factors, including the economic drain of monasticism, eschatological detachment from secular affairs, and cultural rifts caused by theological strife. It is therefore too simple merely to claim that bad morals and a lack of integrity helped the Empire to fall; paradoxically, it appears that the Roman Empire was strongest during its periods of moral decay.