None of the more self-righteous among them ever considers the possibility that people do things for what are very good reasons (lives are complicated).
As for me, I never wanted children because (among other reasons), I knew that based on my family's health history, that I likely would not have a long life. Therefore, I arranged my life so that I could enjoy some retirement years. The only way that I could have done that was foregoing children (which never appealed to me in the first place) and their concomitant costs (college tuition, for instance).
So, I did things according to plan and retired early (and it's been nice, thank you very much).
If some think that I should have fallen into lockstep with what they think is the "correct life" and had children, thereby risking working until the day that I died, well, I'll put it politely: folks, we just don't agree.
You don't have to defend yourself against what some big-shot preacher says. Anyone who actually believes and states publicly that "we grow up by having children" is just plain nuts.