Have you ever been to a baseball game?
They use hand signals quite frequently to determine who will cover what base under what cirumstances, or which pitch to throw, or whether to attempt a pickoff, or any number of other "plays".
Those are not able to be translated by the other team, either. Should they also be outlawed?
Your statement is silly.
What about the kids on the team who AREN'T versed in Spanish, also there's a finite number of things hand signals can mean, one of them isn't for instance, try to break this kids arm when he runs by, which one COULD say in Spanish. Spanish has a lot bigger bandwidth than hand signals for plays. If they want to speak spanish they should pay for everyone on both teams to learn spanish first.