They can still use spells silently. That just means that it can't be defended against as well. If you speak it then it will be blocked. If they don't know what you're doing they can't block it. It is NOT disarming like the Nazis.
And for your info. a lot of those ministries are necessary. Underage magic could be dangerous. The animals are VERY dangerous. The sports games gather attention. Throughout the whole wizarding world they have to make sure no muggles find out about them.
I have to note that the 6th book is the first one to address silent spells, and it is clearly only a well-trained, advanced student that can pull it off, because it requires incredible effort to focus the spell without the verbal componant. Although we have seen low level spells go off quietly and/or with a mere wave of the hand (Snape closing curtains, that kind of thing) the students weren't taught until their sixth year HOW to do it...and it was in the all important 5th year that the Ministry started spoon feeding them the milk of theory over the meat of actual practice.