The Colosseum is in ruins mostly because popes and rulers took stone from it to build the Vatican, not because God let it go to ruin.
I think the point Cal was making is that the Roman Empire was expected to last forever, and Christianity was going to die out quickly.