“I can just imagine what would happen if I wore that shirt (or a button) to work.”
Seriously, that is not appropriate for work.
A P.E. teacher at a school I worked at in a different district would wear his PETA shirt to school. People for the Eating of Tasty Animals. Yes, it generated some negative comments. He was never told to not wear the shirt to school.