I’ve been in Temple and worn one. It was expected even if you were not Jewish. It’s a sign of respect.
If I went to a Buddhist temple in China I was expected to burn incense to Buddha. That is also a sign of respect to Buddhist.
When I lived in China I just didn’t go to a Buddhist temple.
I would politely refuse to go. I’m not anti-Semitic I just refuse to wear a symbol of another religion.