I'm afraid that my husband and I are in the midst of doing exactly what you described for the simple reason that now we can afford to spend our money on ourselves, rather than tuition and other kid things. Also we found the perfect piece of property which demands a large home to make the most of the property. It's just a good long term investment. Maybe we're nuts, but I can't wait until the house is finished. It's on fifteen acres at the end of a private road with 700' of lake front (a small bass fishing lake).
Let me know how it goes! I like talking about building homes. I'm a big Bob Vila/This Old House/Hometime fan.