So, St. Augustine was a hardliner on this one. :) Well, then I disagree with Him too. I agree with the principle of what you are saying. I would phrase it: "God's elect are God's elect no matter what. Since we do not earn our salvation, then no baby can be said to have failed."
Yes, God's elect cannot fail in the end, whoever those might be. I suppose, according to the above, NO ONE can be said to have "failed"... However, on subject, theologians have debated over the place for infants PRECISELY because hell is meant for those who choose to reject God - AND heaven is meant for those who choose to believe in Christ. Since God desires all men to be saved, I would agree with most theologians that God is not bound by the sacrament of Baptism and will not condemn infants to hell - although they also may be kept from perfect union with God, as well. We rely on God's mercy in these cases.
Regards