First, Darwin did not coin this phrase, and as a matter of fact, hated it.
Second, the phrase, when properly understood, refers to a given environment. What is most fit in one environment can be least fit in another.
Moreover, the idea that most fit refers to a characteristic of brains reflects the biases of the reader. Fitness can be the result of any combination of characteristics that allow an organism to reproduce most successfully in a given environment.
More than that... The phrase "survival of the fittest" was actually coined by Herbert Spencer in his Principles of Biology (full of his characteristically useless and turgid broad generalizations) published in 1864. HOWEVER, Spencer had actually formulated his own general theory of evolution, an specifically his theories of social evolution, LONG BEFORE he was aware of Darwin's ideas. And of course it was on Spencer, not Darwin, that "social Darwinism" and most other like abuses of evolution were based.
Now, here's the interesting part. Spencer claimed to have derived his principles of evolution FROM THE LAWS OF THERMODYNAMICS. All the creationists who have for years been touting the laws of thermodynamics as a (supposed) refutation of evolution have actually been touting the "real" scientific root cause (if you insist on that fallacy) of genocide and racism!