Would you say, "I'm not going to force my children to learn math or science - let them grow up and choose for themselves!"? When children aren't given some sort of proper foundation upon which to consider a choice, they will be unable to make a reasoned choice because they are uneducated. Whether it's math or theology. And it will be a hard slog for them to catch up in either case, because they won't have the framework which is really best learned in ones youth.
Also -- the only people who ever say this are nonbelievers, fallen-away (and feeling a wee bit guilty), or atheists. If you really believe in whatever your particular brand of religion is, whether Christian or Buddhist or Judaism or whatever, then you believe that that is the path to eternal life and what God wants you to do. Why on earth would you not teach your children that? Might as well not teach them to read, or brush their teeth.
Finally -- when you don't give your kids any religious instruction, they do VERY weird stuff when they reach their rebellious teenage years. Example: I know two families who do not know each other, have never met, and live in different parts of the country. In each case the parents were nonbelievers -- in one case actual atheists, in the other lukewarm Unitarian/Universalists (if there can be such a thing - but they hardly ever went to church). In BOTH cases the child became a Wiccan Priestess.
What he’s really after is to convert your children to HIS faith in atheism.
Some atheists are supremacist in their advocacy and will declare that all (other) religion should be abandoned (and even prohibited).