That sounds reasonable. But I think a lot of DNA variation doesn't really manifest itself outwardly. For instance, two DNA divergent chimps might look very similar. So not much to select each other for or against.
However, I do agree that aggressive humans were pretty good at "trimming" divergent branches (killing strangers.) Humans are effective and practiced killers of themselves.