You’re right, it’s not a violation of the 1st Amendment. It might be a violation of the Civil Rights Act that prohibits discrimination based on, among other things, religion.
I think the issue is more about “being deprived of property” without due process for something you said”.