The federal government's own website confirms this: "The IRS will hold back the amount of the fee from any future tax refunds. There are no liens, levies, or criminal penalties for failing to pay the fee." Note, however, that future tax planning must be done with equal care because future refunds can be confiscated to cover an earlier penalty.
This suggests that you can avoid the penalty by planning your taxes (e.g., by making lower estimated payments) so you will be owed no refund from which it can be deducted. It's important to ensure that you pay enough during the year to equal either 90% of this year's tax bill or 100% of the previous year's, as there is a separate penalty for too little withholding. This means that if you expect to owe $4,000 in federal income tax, you must pay between $3,600 and $4,000 during the year as withholding and/or estimated tax (or no less than your total bill for the previous year) and pay any balance when you file your return.