Yes, excellent point. CID agents are assigned only cases where they believe criminal activity has gone on. A regular IRS agent would be on cases where they do not believe fraud/crimes exist, but if it is uncovered that it appears to be, it woudl be turned over to CID.
In either case I’d still get a tax attorney. One that deals with the IRS.
The IRS generally doesn’t go after you criminally unless you do something to piss them off (like join an anti-tax group) or you are a celebrity and they want to make an example out of you (like Richard Hatch).
If you refuse to pay they will put liens on your property or paycheck. Generally the IRS just wants the money. They can’t get it from you if you are in jail.