Your "class envy" argument might carry weight if the people in HOAs were defined by having it. Wealth doesn't buy taste. Have you seen the state of the fine arts these days? HOAs are a method of hiring out taste decisions to a committee.
In SoCal, you have a hard time finding a neighborhood newer than 20 years old that DOESN"T have an HOA involved in some way, and I would hardly classify the majority of those as being bastions of anything approaching taste. With the median price being in the high $300,000s, the wealth descriptor becomes problematic, doesn't it?
When we bought, it was a choice to avoid HOAs, because of the numerous associated horror stories from people I know who had been stung by them over incredibly petty reasons. Frankly, we ended up paying more for a similar house because non-HOA houses were in short supply at the time.
Now, I'm in a place where I can operate a woodworking hobby in my garage and my neighbor can grow a truck garden. If either of us is bothered by that, we work it out. We don't trot over to some committee and try to get the other one's pursuits banned by a coalition of uninvolved people from 3 blocks down the way. I like it like that.
In SoCal, you have a hard time finding a neighborhood newer than 20 years old that DOESN"T have an HOA involved in some way