We had a common ancestor, we diverged into two distinct populations, and these two lines combined in a limited fashion by crossbreeds sometime prior to 30,000 years ago when the Neanderthal died out.
Yes, usually fertility between two different species is dependent upon a recent common ancestor, which implies/demands a certain level of genetic similarity. The big barrier to fertile cross breeds is when there is a different number of chromosomes in the parents. Not impossible to overcome, but it does increase infertility.
Similarly, Wolves and Coyotes share a recent common ancestor, AND they came from two distinct lines that have bred together years ago. Coyotes and Wolves developed into two distinct species, but two species that can and DO produce fertile offspring. One can, through DNA analysis, estimate the amount of Wolf ancestry in your average Coyote population - which will show how prevalent these crossbreedings were (or are).
When you all come up with a pure bred Neanderthal remains, then just maybe you all can have credibility there were crossbreeds in the long ago past. Until then it is all pontification called 'scientific methodology'. Now I am NOT saying there was never ever crossbreeding that took place. Just that this notion of a Neanderthal is poppycock.