Nonsense. Sue them, the complainers and the HOA into oblivion for interference with their rights as homeowners and Americans
I for one would not want to live there anymore. So I am going to make them pay me far more than my home is worth. There would be no agreement that I wouldn’t sue afterwards. Thus suing would still be an option, and I would exercise it.