You must understand the culture under which they arose, ConservingFreedom. Promiscuity was also strongly frowned upon, and while both it and prostitution existed, they were not sanctioned, but were instead censured. They could easily make the case for them being contrary to a wholesome culture.
For example, do you want an adult book store to open next to your kids’/grandkids’ elementary school?
No, nor a rendering plant - which can be addressed through zoning rather than banning.