In the case you describe, prices must go up. The only way prices don't go up under the so-called "Fair Tax" is if employers reduce your gross pay to your current net pay. And if employers reduce your gross pay to your current net pay and prices don't go up, you are in the same position you are today.
You are correct but I think a little clarification would be helpful. The part about prices going up should explain that the raw prices themselves don't go up as the employers costs have remained the same. The prices + the tax causes the end price to increase by the amount of the tax, the same as is done by the states and local governments now. The worker receiving his gross pay negates the harm.