Actually, you could abolish (or reduce to a skeleton) the IRS if you went with a flat tax. I’ve stated here before why the Fair Tax has some issues for me, but it you had a true Flat Tax, any entity that pays out money would simply hold that percentage (e.g., 17% as suggested by Friedman) back and remit to the GAO. Your income tax return could be a postcard.
We need to get clear about what we're talking about. From here:
DEFINITION of 'Internal Revenue Service - IRS'
A United States government agency that is responsible for the collection and enforcement of taxes. The IRS was established in 1862 by President Lincoln and operates under the authority of the United States Department of the Treasury. It is primarily engaged in the collection of individual income taxes and employment taxes, but also handles corporate, gift, excise and estate taxes.