Well, to be brutally frank, I'd feel the same way if I was a homebuyer. I lived in an apt complex that went Section 8. Place turned turned into a dump almost overnight. Difference was that I could break my lease.
I'd talk to the BBB about the builder. What he did may have been legal, but it certainly was unethical. And legally, if anything, I'd think that something like "Proximity to Sec 8 Housing" would be required under disclosure when you bought the place.
Good Luck. Remember that I'm not a lawyer, nor a real estate agent. I didn't even stay at a Holiday Inn last night.
This all took place twenty years ago. We took it to court as a class-action suit, but the judge did not find in our favor. Water under the bridge now. I just want to somehow, somehow sell this place and get out.