What I said was that a smaller brain is not a disadvantage for survival in a society where most survival functions are performed at a level above the individual. This has nothing to do with selection of mates or anything else. For example, consider a group of nomadic tribesmen who need to devote 80% of their cognitive resources to survival (hunting, avoiding predators, finding shelter, making warm clothing, etc.), and their group is small enough that everyone must contribute to survival. In that group, children with smaller brains--for whom the 80% of the brain dedicated to survival is only equivalent to someone else's 40%--would not survive.
As humans have become more socialized, those survival functions became specialized to certain members of the group. For example, I don't raise and kill my own meat; I go to a butcher who kills the animals which, in turn, are raised by ranchers. Same thing with clothes: even if someone learns how to make clothes, she doesn't start the process by growing the cotton in her backyard--she buys the fabric already made. And so on. Since survival functions are now performed by specialized members of society, and each of those specialized members is responsible for only a tiny part of a survival function, it is no longer necessary for each person to maintain the brain power to survive individually or in small groups. Hence, average individual brain size has become smaller with no effect on species survival.
The other side of this coin is that larger brains have somewhat of a survival disadvantage. An infant with a large brain might have trouble exiting the birth canal--not a problem with today's medicine, but as little as a century or so ago was a death sentence for both mother and child. It takes more energy to maintain a larger brain, which can be a problem when food is scarce. With selective pressures to maintain a large brain eased because of relegation of survival functions to a large group rather than a handful of individuals, those with slightly smaller brains have a survival advantage.
You also said "size of the human brain has been shrinking". It's one thing to say that small-brained people walk among us and live out their lives: all kinds of people do. Do I need some evolutionary "explanation" for that? No.
This has nothing to do with selection of mates or anything else.
Come now, if women didn't prefer small-headed men then how do you imagine human noggins are shrinking? So it must be that they do prefer them. But despite the assurances of science, I find it hard to believe that there is a general preference for small-headed men among women, even if it is subconscious. As far as I know, Zip the Pinhead wasn't successful with the ladies. But no doubt an evolutionist could argue that Zip was an isolated case and provide numerous examples of pinheads who are reproductively successful and have pinheaded children and so on, etc., etc.
It takes more energy to maintain a larger brain, which can be a problem when food is scarce.
Indeed, as we all know, smart people are the first to starve when times get tough.
those with slightly smaller brains have a survival advantage.
You started out by saying that small brains are "not a disadvantage" and this assertion slowly but surely evolved into small brains are an advantage. Such is the nature of these Darwinistic explanations, it makes no difference if a trait is advantageous, disadvantageous, or neither: the fantastical natural selection narratives people fantasize about simply must be true no matter what the case.