Being a scientist, and therefore a questioning skeptic, I have to ask 1) is this true, since I have not seen this claim before; and 2) whether the soy consumption causes a reduced brain weight, or is the reduction in brain mass a result of the vegetarian diet that usually accompanies increased consumption of soy?
Personally, I feel that if the idea of eating something nauseates me, then I'm better off avoiding it. Therefore, no soy for me.
Actually, all the studies that I am aware of had no relation to vegetarianism; they have been done in places where most people are not vegetarians but still eat varying amounts of tofu. There are two competing bioactive components in soy that could legitimately interfere with brain tissue that are being looked at, but genistein is the likely culprit.
People who consume tofu daily for decades show both measurably reduced cognitive ability relative to the general population and reduced brain weight in autopsies. The brain has a slow but steady regenerative capability, something that was assumed not to be the case until recently; destroyed neurons are replaced in many cases. There is evidence that genistein impedes this repair function (using an action similar to how it impedes the growth of some types of tumors), either blocking or reducing the "regrowth" capability of the central nervous system depending on the amount of soy consumed. Over many years of nearly continuous consumption the un-regrown sections of natural microscopic damage accumulate, whereas in most (non-soy eating) people this type of damage is repaired. It does take a couple decades before there is measurable deviation from the general population cognitively, and it is a smooth divergence so it often isn't noticed or attributed to other things.
So it isn't that soy destroys the brain, but more that it interferes with the cellular repair mechanisms of CNS tissue, allowing damage from other sources to accumulate. You can Google papers published in a few different countries on various aspects of this. Most of the vegetarians I know (who are pretty savvy about this stuff) switched over to similar types of grain protein when papers like this started to surface and eliminated soy from their diets. Not worth the risk to these people, particularly in light of other studies showing other systemic problems linked to soy.