It's a pretty minor problem in Church and if you really did impose their immediate elimination, a lot of people will be alienated. I go to a very conservative NO that does have the female servers too and it appears to work ok. They have lots of servers at Mass, up to 10 or 12 (boys and girls) in a small church and they all do a good job.
If we ever get the Universal indult, it will be interesting how this issue is treated in Tridentine rite. Here, one can argue easily it's for boys only.
The modernists treated the faith of so many with contempt by arbitrarily imposing so many changes. Orthodox Catholics should allow charity to take precedence if the matter is not of sin and with time try to restore the better liturgical choices.
The "why things will never get better" attitude strikes. Our Holy Fathers in the past were not "charitable" as you call it with the enemies of faith, and that is exactly what girl altar boy supporters are. Nor did they dilly-dally and wait for people to come about to their point of view. If the Church has a spot or blemish, you should immediately cleanse it if it is in your power.