Another hypothesis is exactly what is happening today. Wealthy/productive people stopped having kids and the lowest bred like rabbits. Obviously many exceptions exist.
Some of the Emperors even tried ordering people to have kids:
“ In 18 BC the Emperor Augustus enacted laws to encourage marriage and having children, including establishing adultery as a crime. These laws were unpopular and it isn’t certain if they were very effective. They did, however, lead to his own daughter, Julia, being banished to a small island.”
In the same vein
Wealthy people bought up all the middle class farmer land in Italy and started farming it with slave labor (think modern day illegal immigrants) thus hollowing out the the Roman middle class which provided the bulk of the legionaries under the Republic and the early empire. So the Legions started being more and more filled with non Roman citizens. Those new Legionaries were more loyal to their commanders then then any notion of Rome or Citizenship.
The results was political faction (like current US Democrats) who were only interested in advancing their own political power with zero regard for any damage they did the Empire as a whole