You are misunderstanding what I am saying. Sin has distorted us and all Creation. Because our nature is distorted by the ancestral sin, not as a punishment from God on us, by the way, rather than having the "innate" potential to respond fully to God's grace, we have a propensity to reject grace and sin more, thus further distorting ourselves and all creation for that matter and instead of becoming more like God, we beome increasingly unlike God. Baptism doesn't clean up the original distortion (and I understand that the Latin Church doesn't say it does either) but rather empowers us to respond more fully to grace, if we choose, than we might otherwise.
Our conception as distorted images of God is an inherited condition (with admittedly serious implications), not a punishment for a sin we didn't commit or a result of a guilt which isn't ours.
Baptism is not empowerment, but sanctification.
"Christ also loved the church and delivered himself up for it: that he might sanctify it, cleansing it by the laver of water in the word of life." (Ephesians 5.25-26)
"For as many of you as have been baptized in Christ have put on Christ." (Galatians 3.27)
"Know you not that all we who are baptized in Christ Jesus are baptized in his death? For we are buried together with him by baptism into death: that, as Christ is risen from the dead by the glory of the Father, so we also may walk in newness of life. For if we have been planted together in the likeness of his death, we shall be also in the likeness of his resurrection. Knowing this, that our old man is crucified with him, that the body of sin may be destroyed, to the end that we may serve sin no longer." (Romans 6.3-6)
"Do penance: and be baptized every one of you in the name of Jesus Christ, for the remission of your sins. And you shall receive the gift of the Holy Ghost. For the promise is to you and to your children and to all that are far off, whomsoever the Lord our God shall call." (Acts 2.38-39)
Those who are not Baptised are still under the power of the devil.
"Know you not that to whom you yield yourselves servants to obey, his servants you are whom you obey, whether it be of sin unto death or of obedience unto justice. But thanks be to God, that you were the servants of sin but have obeyed from the heart unto that form of doctrine into which you have been delivered. Being then freed from sin, we have been made servants of justice." (Romans 6.16-18)
If Baptism is simply empowerment, than sanctification is the work of man, rather than of God. God puts the means at our disposal, and then leaves us to respond to complete the task. But St. Paul says: "God is he that justifieth" (Romans 8.33). And if Baptism is empowerment to more fully respond to grace than we might otherwise, is there another "Way" (St. John 14.6) to eternal life besides putting on Christ, or another way to Christ besides Baptism?
My problem is this: on the one hand, you say that the infants already have holiness and justice at their conception (am I reading you correctly?) - on the other, that they lack even the possibility of divinization until they are baptized. To me, that appears contradictory - when I talk about the original holiness and justice of Adam, I am talking about him being originally divinized, made to the "image and likeness" of the Holy Trinity (Gen. 1:26). No one can be holy or just except by participation in the nature (2 Pet. 1:4) of the One who "is good" (Mt. 19:17), so either infants are conceived in a state of being like to God (holy and righteous) or of being unlike to God (that is, "sinners" Rom. 5:19). Am I missing something? I really don't see how your view isn't self-contradictory.
For I have shown that the Son of God did not then begin to exist, being with the Father from the beginning; but when He became incarnate, and was made man, He commenced afresh the long line of human beings, and furnished us, in a brief, comprehensive manner, with salvation; so that what we had lost in Adam -- namely, to be according to the image and likeness of God -- that we might recover in Christ Jesus. ... For as it was not possible that the man who had once for all been conquered, and who had been destroyed through disobedience, could reform himself, and obtain the prize of victory; and as it was also impossible that he could attain to salvation who had fallen under the power of sin -- the Son effected both these things, being the Word of God, descending from the Father, becoming incarnate, stooping low, even to death, and consummating the arranged plan of our salvation, upon whom [Paul], exhorting us unhesitatingly to believe, again says, "Who shall ascend into heaven? that is, to bring down Christ; or who shall descend into the deep? that is, to liberate Christ again from the dead." ... For as by the disobedience of the one man who was originally moulded from virgin soil, the many were made sinners, and forfeited life; so was it necessary that, by the obedience of one man, who was originally born from a virgin, many should be justified and receive salvation.When, however, the Word of God became flesh, He confirmed both these: for He both showed forth the image truly, since He became Himself what was His image; and He re-established the similitude after a sure manner, by assimilating man to the invisible Father through means of the visible Word. ... But inasmuch as it was by these things that we disobeyed God, and did not give credit to His word, so was it also by these same that He brought in obedience and consent as respects His Word; by which things He clearly shows forth God Himself, whom indeed we had offended in the first Adam, when he did not perform His commandment. In the second Adam, however, we are reconciled, being made obedient even unto death. For we were debtors to none other but to Him whose commandment we had transgressed at the beginning. (St. Irenaeus, Against Heresies III, 18; V, 16)