I wouldn't care either. But you are not part of their congregation. For a chaplain, every religious person there is part of his congregation.
A congregation necessarily consists of people of similar belief or ethnicity.