About your assets, why did you accumulate so much outside of tax deferred accounts? You say you spent a lifetime saving this money. If you had used IRA's or something similar you would have had a much more rapid compounding of returns than you did outside of these. I've found the best combination is about 50-50 taxable and tax deferred. Most of the problems I encounter is with people who have too much in tax deferred accounts.
I've have a settlement in a personal injury suit that gave me a tax-free structured settlement annuity. These things will be greatly affected by the "FairTax" - and I suppose that there are lots of people similarly situated.
I've also got lots of non-discresionary expenses that are favorably treated in the current tax law, that are disregarded by the "FairTax", but were enacted through years of experience with trying to make the income tax fairer to people in special situations. Got lots of medical expenses - ok, deduct them from you income. What's wrong with that? You want to keep the taxpayer alive, right? If only to prick his piggy bank.