The problem with that are homosexuals and pedophile homosexual predators, transsexuals and transgenders. You get someone who looks like a woman, has breasts and a penis.
There's nothing stopping a homosexual pedophile from entering a men's room, uncontroversially, right now. At least genitalia signs provide a clear answer to the trans freaks who claim the right to use the ladies' room because they're really a woman with a penis.