There were other factors, of course. The establishment of Constantinople split the empire into administrable regions once it became obvious to Constantine and others that it was too large geographically to be administered from Rome. That in turn revivified the already strong Greek influence within the empire. It also split the revenue stream, a good deal of which was, actually, filling Church coffers but not (my contention now) enough to fully explain the atrophy of Roman government.
In the end it was the influx of migratory peoples that changed the empire the most, a wild cascade of tribes that first surrounded, then invested, and then infiltrated the Roman body politic. The real damage was already done by the time the Vandals swept through Spain and into the breadbasket of Rome, which was northern Africa. Once the food supply was controlled by the barbarians it was all over, at least for a century or so until Justinian's great general Belisarius threw the Vandals out of possession. Procopius chronicles what happened next in the Gothic Wars, when the Greeks attempted to wrest Rome from the Goths. We are here into the hideously misnamed "Dark Ages." Gibbon takes us through this and the next nearly thousand years in the Decline and Fall. Some of the best stuff I've ever read, and highly, highly recommended.
Don't tell me your statement of hideously misnamed Dark Ages will be twisted like the ...there was no Holocaust...just something made up by the Jews. Why do I feel I won't be surprised. :)