That's where some kind of insurmountable limit on budgets and authority would come in. Such limits would aid truly conservative administrations in shrinking the monster, while hindering the National Socialists when they, regrettably, are temporarily in office. Plus it would give conservative administrations some cover when fighting with those who would like to grow the department, by saying "Shucks, I'd like to turn the department into a branch office of the DNC like you're asking, with the power to summarily execute conservatives, but regulation 53-48XYZ forbids it."
That's why I think we need to get away from using the "blank check" Commerce Clause, and start specifically authorizing these agencies by amendment. That's the only kind of insurmountable limit Congress will understand.