I did exactly that on a cross country flight, east to west, just before the pandemic hit. I had the window and the two passengers to my right could not hide their disgust. One of them started mumbling stuff about Trump after he saw my hat. But to my surprise one of the flight attendants (female) who came by to serve snacks saw my hat and was delighted.. gave me a big smile and a thumbs up ! That was great. The passengers to my right were looking in disbelief.
Saw a post here recently about the opposite happening with a male flight attendant being inconsiderate/rude throughout the flight on Southwest.