Your assertion that three out of four Catholics quit attending church is simply false.
80%+ of able-bodied, prime-of-their-life Catholics attended Church, perhaps. Few received Eucharist regularly however. Today, according to surveys, about 40% of all Catholics, including the aged and infirmed, attend mass. The surveys I've seen show the drop-off at stopping about the mid-80's, until the priest scandals. THen the # dropped to 27%, before rebounding back to over 40%.
Personally I suspect this is the greatest reason for the drop-off that did occur: It used to be that the "unworthy" attended mass weekly, and skipped just communion. But so did most everyone else. Then, after Vatican 2 made communion more acceptable (less fasting, more masses scheduled, big push on everyone should receive who can, etc.), those who did not feel worthy felt singled out. The expectation was that if you showed up, you should receive communion, and if you couldn't, then you were an outcast.
Then came Humanae Vita (sp?), and millions of people who had expected that the Church's position against the birth control pill was obsolete, were stunned to find that the church considered them to be in mortal sin.
40%? Show me, please. I think it's more like 17%. You also fail to make note of the fact that most communicants today no longer believe in the Real Presence. And more people have the attitude that sin is no big deal. The consequence is a higher ratio of communicants at Mass--but a lower number of overall attendees.