"That may have been a contributing factor, but it wasn't the cause by any means. I've also heard it attributed to military overextension, economic woes, the cost of the 'bread and circuses' welfare state that developed, growing military resistance in outlying areas and increased military prowess and strength of the various 'barbarians', hedonistic decadance and self absorption among the leaders resulting in almost no vision for, or respect from, the people, etc. These are all probably factors but could more realisitcally looked at as being the visible symptoms (not the cause) of a culture in decline.
A strong moral people can withstand all kinds of severe hardships and setbacks and still survive, even grow stronger, in the presence of them. An immoral people falls to whomever or whatever seeks their demise.