Yes, you are paying for them when they default on their loans because they were so stupid to buy way more than they need or afford. Also, the minute a mcmansion goes up in your neighborhood, the tax appraisal district jumps all over it and raises your property tax.
Not a problem in my small town.