IMHO, baptism needs to be followed by bringing the child up in the nurture and the admonitions of the Lord. If this doesn’t happen, how can the child be filled with the HS? The child will probably not grow up to know God so the Spirit of God doesn’t dwell in him or her. A baby doesn’t know anything about God and if parents don’t teach this child, or bring him/her to church, then what?
I do believe if God intends for you to be one of his it will happen, some roads are rougher than others. Also, we are told to preach The Gospel to all nations.
I do follow your point though. It is much better for a child to grow up in a Christian church, but that does not guarantee that child will be a Christian even if they were baptized as an infant. That child could easily grow up, stay in the same church all their life and never truly believe The Gospel.