The Defense budget is 663 billion, or 1/5 of the entire federal government. That is hardly trivial.
There are 4 million Federal employees. If their average salary was 100,000 and we fired every single one of them, we would cut 400 billion, which is only 2/3 of the defense budget. The three monsters (20% of the budget) are defense, medicare/caid, and social security. If you add in debt interest, those four payments would equal 70% of the budget. Those are the four things that have to be looked at first if you actually want to reduce government spending.
The difference is that defense is needed. A Dept of Education, or HHS is not.