The human brain is almost certainly the result of sexual selection. Once you have an organized society, wealth can be achieved by brainpower more than by muscle. And it is wealth that attracts the women.
I agree with you but I remember a 20/20 or Dateline or some other such program about the issue of short men.
They had women select men through a one way glass and the women ALWAYS went with the taller man even if he was unemployed/"underemployed" or undereducated. The short men were even built up to be wealthy, educated, etc. but they still lost out to the tall guys.
There just seems to be something hardwired into women to not select guys they have to look down on, literally that is.