They do not start out physically female, they start out undifferentiated. All embryos start out with a "bud" in the genital region that develops into characteristically male or characteristically female genitalia, barring some genetic problem or androgen insensitivity, in which case that "bud" can fail to develop, leaving no external genitalia and no internal genitalia, or it can partially develop with aspects of both male and female, rarely.
Let me put it this way: all embryos develop into females unless the early gonads produce testosterone which act on androgen receptors throughout the body to effect male development.
Have they discovered what causes androgen insensitivity?