There are modern men of varying heights -- Danny DeVito's skeleton may well more resemble a neanderthal's than it would Michael Jordan's. I wonder how much diversity in height and build is allowed for neanderthals?
As Vade pointed out, it's not just the height and build, it's the ratio of bone length to cross-section, the shape and build of the muscle attachment points the shape of the skull and jaw, the angle at which the spinal cord enters the skull, the shape of the sinus cavities, etc. There are numerous morphological differences between Homo Sapien and Homo Neanderthalensis. A trained anthropologist or even anatomist can quickly tell the difference between human and Neanderthal bones.