Absolutely not. On the other hand, neither should he be able to shut up the parishioners if they decide to proselytize to him.
So, in this case, the fact that the one guy engages in conduct that the vast majority find highly offensive does not give the majority the power to silence him.
That is my larger point- rights are not subject to a veto by the majority, no matter how obnoxious the enjoyment of those rights by the minority might happen to be.