In my mind, its a valid argument. Healthiness/Wholeness is a legitimate cultural concern. We forbid blood donations from those in countries with mad cow disease even though the probability of infection is in the multi-millions to one.
Drugs are a public health threat, if at all (nobody ever got a communicable disease from pot), because of their illegality and consequent behaviors like sharing of hard-to-get-because-illegal needles. And prostitution is no more a public health threat than noncommercial promiscuity - should we criminalize that?
You must understand the culture under which they arose, ConservingFreedom. Promiscuity was also strongly frowned upon, and while both it and prostitution existed, they were not sanctioned, but were instead censured. They could easily make the case for them being contrary to a wholesome culture.
For example, do you want an adult book store to open next to your kids’/grandkids’ elementary school?