This is a good point. A recent book out about the development of language asserts something along this line. Some monkeys have different screams designed to alert their fellows to different dangers. Lions get a different scream from them than other predators. Is this a language? I think not. It's communication, but not language.
That's my sense of it. Pointing to a wooly mammoth skeleton and grunting three times while pointing beyond the next hill may well communicate to the other Neanderthals that there are three wooly mammoths waiting over the hill to be hunted done for dinner, but lacking abstraction, syntax, structure, and so forth, it is hardly language. Or so it seems to me.