Bingo.
The chromosomal approach is flawed both biologically and sociologically. There are conditions which cause persons with an XY chromosome pair to develop morphologically as females, while what rightly disturbs normal people in circumstances like bathrooms and locker-rooms is encountering people with complimentary genitalia to their own.
How often does that happen? 1 out of 10 million births?