That's what I thought until I actually read Mark Steyn's book which explained the math of demographics in everyday language. Civilizations aren't doomed just because their birth rates dip below replacement rate, but they are most certainly doomed when their birth rates stay below replacement rate for two or three generations. By that time, there is no way for and elderly, childless population to even sustain itself, let alone breed its way out of decline. And since the human nature will not permit a power vacuum, younger populations will inevitably sweep in to plunder the wealth that can no longer be defended -- and to do with the remaining inhabitants as they will.
Today, those civilizations who LIKE children are not the good guys.