There's nothing wrong with civilized people agreeing to a common set of rules.
What people have a problem with is that these HOA's can be co-opted by people who abuse the power to settle personal vendettas, forclose on people's homes to eliminate a minor inconvenience, and frequently change the rules of the game artbitrarily or specifically to oust a percieved enemy. You may not have been on the wrong end of a HOA but many on here have, and it wasn't because they violated rules they agreed to.
HOA's are limited. They can go to court and ask a judge to enforce rules. They cannot forclose or kick people out. They are elected by the people they serve. Actually for a HOA to take action, it takes a long time to get anything done. That is why most do very little until something is egregious.