If you’re talking about overtly political “art” that relies more upon the bio of the artist than any aesthetic assessment of the work (Mapplethorpe, et al.), I’d agree with you.
However, there are subtleties that people who have not studied the matter in depth will not necessarily recognize, in color palette, materials, composition and choice of subject matter that validate certain works of art that are difficult to grasp for the general public.
Music and literature go through the same vetting process and generate controversy, but not nearly to the level of visual art. People who know nothing about the subject have strong opinions about what they “like” and what they don’t like. Others judge paintings by whether or not they’ll go with their furniture and draperies. These are perfectly valid reasons to buy something if an individual is so inclined. But, it doesn’t make it good and it doesn’t make it “art.”
It seems to me that you are letting so-called experts define for you what art is, and to me, that is as absurd as letting someone else tell you whether or not what you just ate tasted good.
"Wow! That tasted like crap!"
"No, no... that was a very important piece by an excellent chef."
"Really? Can I have some more then?"
Rockwell's biggest problem with his critics was his popular appeal.