All tax systems require reporting, collection and enforcement. Eliminate IRS? Sure, but replace it with some other organization doing the same thing.
You're exaclty right. Without meaningful spending cuts, it will be an empty gesture.