Simply cut the funding to all those departments. Starve the beast. Or move the agencies away from Washington District of Caligula. No reason to have them all concentrated in one region unless you want to herd them in for the nuclear kill (the only good reason to keep them where they are).
A good spot for the Department of the Interior would be Idaho or Montana. Not too many Pygmy cannibals will want to move to those rural enclaves. And the new FBI building should be built in Nome Alaska. /spit
I guess you don’t realize that the vast majority of Federal employees do not live or work in the District of Columbia. The Department of the Interior has facilities and personnel in every state already.