There is no such thing as physical gender. It’s a grammatical category. People are sexed, not “gendered.” The whole notion of “gender” as opposed to sex is an invention of radicals, based on their argument that maleness and femaleness are social constructs and not biological realities.
You’re correct of course, and another reason why language matters.