You have an excellent point. I specifically looked in neighborhoods with no sort of HOA or convenant when I bought my first house a number of years ago so that nobody could tell me what I can or can't (legally) do with my property.
I did the same thing. I paid an attorney specifically to go over the agreements.