Those guys are vastly outnumbered.Those women can take care of themselves.
In all cased I was escorted to my work site and protected by correction officers (never call them guards).
At the Women's Correction Center I worked at there was one person with male facial hair I saw working with the women. I asked the correction officer what was with the. The officer told me that the person in question had been castrated and for his/her own protection was not allowed in the male prisons. The officer said that the person didn't have an easy time at the women's prison.
Now as to the whole bathroom question. I think that it is being overblown for political and litigation reasons. There are lots of public places that have men's, women's, and either handicapped or “family” restrooms.
In Europe I slept in a number of hostels, which were like dorms with large rooms of bunk beds. Several of them were co-ed and the showers and bathrooms in there places were kind of interesting. The all had lockable doors leading to a private changing area and from there a door leading to a private shower. Some of these hostels were run by nuns and some were part of religious group settings. Others were privately run.
The point is that if you have the right kind of facilities that allow for privacy, as opposed to communal or group showers, or group changing areas, this becomes a non-issue.
Those states that are hell-bound to want to provide transgender folks to use the bathroom identified with their non-biological self, should pass building codes that require different kinds of men's and women's changing rooms and shower facilities and bathrooms. This will make new construction very expensive, but if the jurisdiction wants to be transgender friendly, that is their choice. If they want to be real friendly, then can offer grants to help with remodeling of schools, hotels, YWCA’s, parks buildings, etc.