You're conveniently forgetting "where" the Chaplain intended to wear his uniform.
It's what the Navy commissioned him to do in the first place.
Now if the Navy wants to do away with the Chaplains, it has the legal right to do so, but so long as his uniform requires religious symbols on it, everytime he appears in it he is doing so for religious purposes.