That's a good question, how they vanished completely. I'm thinking it had to be a serious plague to which they were vulnerable while our guys were not. Maybe related to cannibalism.
Otherwise---such gradual marginalization that eventually they were left to small isolated areas where inbreeding and its nasty social dynamics did them in. I think they had some sort of culture that broke down, and that alone can kill off a community, no plague necessary.
Still the question remains, since 100% gone (after that successful long run) does seem awfully strange. It's just that we are thinking in terms of the "last man" among them, and actually we have to think of the last community. Once the "community" has collapsed, from a social unit to mere numbers of individuals (every man for himself), the process of extinction speeds up real good.