You certainly are in the minority. Check this link:
http://www.fairtax.org/pdfs/Homebuilders_will_benefit.pdf
and then the opinions of 75 economists:
http://www.fairtax.org/pdfs/Open_Letter_President.pdf
I have heard homebuilders on talk radio shows who are very much in favor of the fair tax.
I'll also bet you dollars to donuts that the final version of the fair tax will exempt new homes.