Your inflammatory descriptions aside, you're quite correct -- and the fact is that there was quite an "racial superiority" industry built on that basis. So much so, in fact, that Stephen Jay Gould felt compelled to write his "Mismeasure of Man" to address the issue.
Oh, it goes deeper than that. The "Out of Africa" theory of human origin and migration contains a humdinger, implying that those of lower intelligence remained.