There are studies that suggest that the distribution of the Neanderthal genes to sections of the genome for disease resistance and a few other sections while purged from the rest of the genome suggests the opposite. That there were hybrids but less fertile than average, but the disease resistance their descendants had more than offset the lower fertility.
The lack of Neanderthal mitrochondria, though, may mean that it was all one way genetic interchange - rape by Neanderthals of human women, the male hybrids killed, females kept to increase the human tribe’s numbers. Such a scenario also explains the lack of Neanderthal Y chromosomes in the human population.
Or the male hybrids weren't fertile but the female hybrids were.