You are right. There is a reason fishing lures are often brightly colored, it attracts fish. I wish environmentalists would "follow the science". Very few goldfish survive to that size in the wild because of predation.
Usually when they breed outdoors the ones that survive to breed and grow are the ones with the poorest color, so in short order, in certain bodies of water, you have tons of silvery goldfish doing all the breeding, not bright gold ones. It is those silvery survivors that proliferate and become a nuisance because many of them get big enough to avoid the usual predators.