It has more to do with the homosexual movement than with celibacy actually. I recommend Cardinal Medeiros' 1979 letter to the Vatican outlining the new homosexual subculture which was becoming apparent in the 1970s in Catholic seminaries.
I also believe the hyperincidence of perverts in the diocesan seminaries started well before the 70s, seeing as some of the more eggregious cases -- Geoghan and Shanley, for example -- were already out of seminary and committing their crimes by the 70s, although the perverts became bolder and more apparent as time went on.
What angers me the most, however, is that the bishops had a heads up about these perverts in the mid-80s, yet continued shuttling them about instead of weeding them out.