Unfortunately, my friend, your thinking on this is short-sighted and too narrow in its scope.
Let me explain what I mean, please:
All of that labor you are referencing, no matter its true size, is UTTERLY wasted, inefficient labor.
All of it.
It doesn't produce a single useful product or service.
It doesn't produce one dollar of revenue for government.
Nada.
Might as well just flush all that time and effort right down the toilet.
Freeing up that time, which by any honest estimate is huge, would be a boon to the American people, either in time that could be used to actually produce something of value, or even, hallelujah, to devote more time to their families, their communities, or even to leisure.
Not all benefits have a price tag. Life isn't all about money.
Long term, eliminating the drag on all of us that is created by this inefficient, silly income tax system, would be a positive good.
It may well be wasted effort. But that doesn't change the fact that, if you lay them all off, you risk a depression and re-assigning them to truly productive work takes at least a year, while prices have already gone up by say 30% and stayed there.