This issue with contraceptives is that the hormones are synthetic. The synthetic version of oesterone is “ethinyl oestradiol” and has been found in male river fish to wreak havoc his (ahem) male-ness.
Likewise, the synthetic version of progesterone is called progestin and is a major player in breast cancer.