The remedy would be to provide that a person must have one of a number of characteristics to be called a man, and must have one of a number of other characteristics to be called a woman; a few people would have characteristics that would allow them to be called either. Those people, and those people alone, would be allowed the discretion to declare their sex.
Your solution is quite... vague...