After a few hundred years of being able to segregate public restrooms according to gender, do you think this is actually beyond the scope of "public"?
Do you honestly think the only danger is a women’s restroom is men? Or in a men’s restroom, women?
The only time I have been accosted in a women’s restroom was by another woman/girl of a privileged class who decided she wanted my earrings.
This world is full of weird, dangerous people. That does not magically change when you walk through the door of a public restroom if you can only keep transgendered people out. 99.9% of the time you wouldn’t even recognize them.
Time to get a grip and start using our brains. That’s what God gave them to us for. Although it’s obvious some of us have been short changed in that department.