What if their contract said no Negros?
There are federal and state laws that prohibit that kind of rule.
The HOA has pretty much the same rule making power that the home owner himself has. They can make any rule that is not itself illegal for the home owner to make provided that the basic HOA charter did not prohibit the rule.
If a home owner can prohibit guns on his property the HOA can as well.
For this very reason I would never buy property in a HOA development. I don’t want a HOA telling me I can’t fly a flag or build an out house if I want.
Some people like them because they force an owner to keep up the appearance of his home. Some people really worry about their property being devalued by bad neighbors.