You are correct. The hysteria is unwarranted.
Here's a biological woman.
Which bathroom should she use?
If she's forced to use the bathroom that corresponds to her sex (as opposed to her gender) she's likely to get her ass kicked.
All cross dressing women always have beards.