I mean I don't think there should be altar girls! But, I do think it has a lot to do with parents being afraid of having their sons around pervert priests, and with good reason. Go to Roman Catholic Faithful and read the sad letters from parents betrayed by their priests, bishops, and even their own community. That has to be atoned for and corrected. V's wife.
While you are right, the drop in altar boys has been going on since altar girls were first allowed. I know a couple of boys who stopped serving once the girls served beside them.
I never could understand why the idea of altar girls flew... didn't this used to be the first step in the recruitment of boys for the priesthood?
As an aside, I have a friend who will not allow her son to be an altar boy. She is quite vehement about it and cites the scandal as her reason. Then I reminded her that she doesn't go to Mass anyway, so why would she have her son be an altar boy?