I have never seen any evidence the later empire was less populous than in the time of Augustus or Marcus Aurelius. Or that the “barbarian” tribes across the borders were any more numerous.
For a host of reasons there were fewer men willing or able to fight to defend the empire. In fact, there are historians who claim that the “fall of the empire” was as much a series of revolutions by the inhabitants as it was invasion by outsiders.
In any case, it is obvious there were few willing to fight for what had become an oppressive system. Most of them discovered the hard way that life became much harder for them after the empire fell, but by then it was too late.
I’ve seen a few documentaries that said there was a big population decline among Romans in the last 200 years of the Empire. Per the below link, apparently plagues were to blame:
http://www.unrv.com/empire/roman-population.php