That's a start, but the government agents involved don't care, because it's not their money, it's yours/ours. There should be serious career damage to government employees who do this sort of thing, along with the compensation you suggest.
People act as if "due process" exists when you can bankrupt yourself against the unlimited resources of the government in defense, and that isn't taken into account. In effect, the government, or a petty bureaucrat of the government, can find you guilty and fine you the cost of proving your innocence, or worse, with no risk. And, maybe it's a plus on their resume, too.
While I hate them, I will give the Church of Scientology credit for one thing—they are the only group I know of that out bullied the IRS. For decades the IRS refused to recognize them as a religion for tax exempt status, and claimed they owed hundreds of millions. When the IRS put a tax lien on their HQ, they followed L. Ron Hubbard’s dictum to never defend, always attack. They infiltrated members into the IRS and hired PIs to dig up dirt on everyone at the IRS who worked on the case. They stole massive amounts of documents, and claimed unknown parties just dropped them off. Then they filed multiple lawsuits against not only the IRS, but the employees involved, even demonstrated at their homes. Like all bullies, the IRS is better at dishing it out than taking it. Eventually they waived most of the money “owed” and settled for a one time payment of 12.5 million in exchange for tax exempt status for the church and over 100 subsidiary organizations.