Hmm. Seeing how they promoted the deception that they lived in fear of the creatures in the woods, would they actually dare to chop down the trees required to build their homes?
Heck, it's more basic than that. They could explain away existing structures easily enough. New buildings would be a big problem, but we don't necessarily know that there were any. The bigger problem is how the heck they heated the buildings in the winter. The trees and the look of the place is roughly consistent with a PA latitude, but even at MD or VA latitudes heat would be needed in the winter, and they were not further south than that. Where does the wood come from for heat?
Perhaps it is all part of "Those Logic Problems We Do Not Speak Of".