Human genes and Neanderthal genes are 99.5% identical.
Human genes and chimpanzee genes are 98% identical.
It is estimated that only 1-4% of the DNA of anyone with non-African ancestry is of Neanderthal origins. So of 100 ancestors at the time of gene mixing - some 99 to 96 of them would be human and only 1 to 4 of them Neanderthal.
Thus your idea that the difference between non-Africans (not just “whites” but all Asians, Islanders, Persians, Arabs, Europeans, etc) and African people would be much larger is pure bunk.
First there IS more difference in DNA between Africans and non-Africans than between any other group and the rest of humanity.
Secondly your assertion that Neanderthal DNA was halfway between ours and chimps is absolutely wrong.
Thirdly, mixing in 1 to 4% of something 99.5% identical is not going to add up to a “huge” genetic difference.
A lie does not become truth via repetition.