I agree with you, and I don't think it's far-fetched at all to suppose the Neanderthals would be discriminated against based upon appearance. Mate selection based on all sorts of physical characteristics is widespread in the animal kingdom. Typically mate selection favors breeding with someone similar to one's own appearance, since creatures with a much different appearance may be of the wrong species, and breeding them would be a waste of time!
I think that any love is good lovin'
And so I took what I could get, mmm
Oooh, oooh, she looked at me with big brown eyes
And saidYou ain't seen nothin' yet
B-B-B-Baby, you just ain't seen nothin' yet
Here's something that you never gonna forget
B-B-B-Baby, you just ain't seen nothin' yet