The question is whether or not the pastors First Amendment rights are being violated. They most certainly are.
Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof...
The First Amendment is a PROHIBITION on the federal government, not a granter of power to enforce the First Amendment.