Sorry, what is "despicable" about the idea that clergy shouldn't be an agent of the state, and that communicating with a clergyman in a religious context is protected just as much as communicating with a defense attorney?
If "no establishment of religion" means anything, it ought to protect your right to speak to a minister of your choice without the state horning in on it.