Taller men may seem to have an advantage over the shorter ones in a brawl. That is a myth, a tall guy HATES brawling with a shorter guy, once he has experienced the situation. I am not talking about Marquis of Queensbury rules here, I am talking about survival. All of the points that can kill a man are more easily accessible from below, than from above. Testicles, liver, kidneys, sternum, esophagus, any one of these struck properly, and the guy is going down. Ever watch Joe Frazier work inside? WOW, that smarts.
You are absolutely correct, plus there is the psychological factor.