If the FSP is comprised of the type of anarcho-ideologue loons we see squatting on FreeRepublic, your concern would be moot anyway, as their Jonestown would last about a month.
I don't see how you could possibly be against this. All the libertarians could move to their state, stay out of your hair, and you could stay in California. What's the problem here?