In my years as a chaplain on active duty, I saw exactly one regular occurence that might cause an atheist to wonder. It is as pointed out on this thread...the prayers at required public ceremonies.
(1) One can end all such prayers because they unnecessarily promote a variety of religion each time one is prayed.
(2) One can continue such prayers because they are a tradition in the same way as a “pass and review” in a required ceremony is a tradition. In other words, the religious content is not the point. The point is maintaining the tradition. In this sense, they would be similar to the statue of the 10 commandments placed in the town square in 1851. They are tradition and are not there because of a religious message being imposed on anyone.
I think #2 is a valid explanation of their existence. At the same time, if the prayers are going to be required to take a certain form that avoids religious content, then I would be opposed to giving up my freedom to write the prayer of my choice for the event. In that case, I would support #1.
I think half the problem is the reliance of commanders upon chaplains - who are unavoidably a chimera of church wed to state - as though they are social workers.
A true atheist will consider a prayer to be a pointless and harmless endeavor. I guess those who get upset at a prayer are atheists without a deep faith LOL.