No, I would not permit such a trivial event to force me quit my job.
What if my Boss told me not to wear my Knights of Columbus jacket to work?