i’m not religious either though i follow a Judeo-Christian ethos. i do get a feeling of serenity when i see a creche or menorah. it seems these in your face atheists do not have a lot of faith in their “lack of faith” if they are afraid of any symbol of religion.
I'm a classical music fanatic, so despite my lack of religion, some of my favorite pieces are requiems (Mozart, Berlioz, Verdi, Brahms) and masses (Schubert).