No, this is nothing but media driven misinformation. For men and boys, the phytoestrogens in soy do not appear to have any effect on hormone levels and have not been shown to affect sexual development or fertility.
Isoflavones are dissimilar from estrogen in many ways. Unlike estrogen, isloflavones are tissue selective and can have estrogen-like effects in some tissue but either no effects or antiestrogenic effects in other tissues. Isoflavones are referred to as phytoestrogens because they bind to estrogen receptors and exert estrogen like effects under some conditions. However, the ability of a chemical to bind to hormone receptors, such as the estrogen receptor, won't tell you much about potency or any resulting biological activity.
Women predisposed to breast cancer should not consume soy either because phytoestrogens promote cancer cell growth.
Yes, it is a commonly accepted view that food containing isoflavones, specifically genistein, mimics estrogen and accelerates breast cancer in susceptible women. (Those already with cancer that are estrogen receptor positive). This view, however, ignores Asian women who eat a diet heavy in soybean based foods, and therefore have a higher exposure to genistein, but do not share this same experience with breast cancer.
In Okinawa, soy makes up about 12% of the diet of elder Okinawans yet they experience 80 percent less breast cancer and prostate cancer, and less than half the ovarian cancer and colon cancer, than we do in north America. Go figure.