What is so hard to understand about ‘if you have a dick and balls, you are a guy and if you have a vagina, you are a gal’?
Bingo.
The chromosomal approach is flawed both biologically and sociologically. There are conditions which cause persons with an XY chromosome pair to develop morphologically as females, while what rightly disturbs normal people in circumstances like bathrooms and locker-rooms is encountering people with complimentary genitalia to their own.