Thats is a very astute observation, and so I can tell you are a divorce attorney. I am still head over heels in love with her (emotionally), and I know she is with me. The truth is she cant deal with the guilt, and I cant deal with the shame, even though this happened 16 years ago. It just keeps growing. As my children and cousins start finding out what happened (as they get older) it has this incredible snowball effect that is really overwhelming. At a certain point you have to move on.
You have to live life the best you can.
I'm glad that you and your wife were able to seemingly raise your kids without subjecting them to your pain; you would not believe how many a--holes there are out there who use their kids to get back at the other. I'd say at least 50% of my cases involving custody disputes has this happen.
I wish you well.