You are right, there's no way to get rid of the IRS without getting rid of the income tax. The vast majority of tax law is about what is and what is not income.
A flat tax is only simple for wage earners. And it won't stay simple for long, as somebody will soon get elected by promising to tweak the income tax to reward their voters.
A flat tax is only simple for wage earners.
Elementary school arithmetic.