I think they have the right to assemble but just not to actually discriminate against others
If you cannot exclude people then you are not freely assembling, you are assembling under conditions set by the government. One cannot have it both ways, either we are free to refuse service for any reason, or we may only refuse service for government approved reasons.