Something I read on FR a while back ... a study that showed a close correlation between the number of words a child could understand at age 5, and his or her success is school over the following several years.
Verbal processing is a key skill in civilized life. This is not to say that a person can't have manual skills without verbal processing, but the lack of words limits a person, however manually dexterous, in his access to the goods of civilization.
Keep in mind also that the latest thinking in regard to what drives social creatures to become more intelligent is less the challenges they face in the environment, and more the interpersonal conflict within their own group.
Generally speaking, if you happen to be dangerously stupid, like the kind of guy who might inadvertently lure a saber-tooth back to the clan's cave by dragging a bleeding carcass all the way, then the first ones to notice will be your closest relatives, and their opinions might control which assignment you get on the next woolly mammoth hunt.