One of the main reasons why Fairfield County,CT is so popular is because there are very,very nice towns....just 45 minutes from Grand Central...that have property taxes (and,IIRC,income taxes) much lower than in Nassau County or Westchester County,NY.
Now you’re talking real money. LI is kind of showy, but the real wealth is in Fairfield county. It is hidden in the back country, but it is there.