Even the most distantly related human populations shared a common ancestor within the last 100,000 years; neanderthals diverged as a separate unique population some 500,000 years ago.
In morphology they are distinctly neanderthal, not human.
In DNA they are distinctly unique, and show the evidence that the fossil record supports; that they diverged some five times longer ago than even the most distantly related human populations.
You assumptions of extrapolation of the 1-4% ancestry as being a 1-4% difference and that being the same amount of difference between humans and neanderthals is flawed.
The 1-4% ancestry is based upon markers and does not denote a 1-4% genetic difference between humans and neanderthals. Neither is the genetic or genomic difference between human populations and between human and neanderthals the same amount of difference, but five times less.
In other words, if you found a DNA sequence where two human populations differed by 0.1%, you can expect to see the neanderthal sequence differ by around 0.5%.
In the chart of genetic distance between human populations that I posted, the farthest branching would only be around 20% of the distance between the branching of humans with neanderthals.
For the Neanderthal side that's a serious problem since their remains would have gone "under the ice" numerous times and only a small sample of their quite lengthy history can ever be known.
What we don't know is whether or not any of the Neanderthal line returned to Africa ~ e.g. during those 5,000 year long periods every 100,000 years when the Sahara is in a pluvial and filled with trees, grassy plains, and plentiful game.
We don't know if our type of folks actually made it to Europe during any of the interglacials of those times just before they were destroyed by advancing ice sheets.
At the moment we only have a good grasp of our own line of folks coming from Africa and staying over a major glacial sheet advance by residing in South Central Asia.
There are entirely too many possible points of Neanderthal and proto-modern man to interact over that 500,000 years for us to exclude anything.
Seems to me you're working overtime to emphasize that our DNA "glasses" are one-tenth of one percent empty.
I'm simply pointing to the fact: the DNA-glass is 99.9% "full" of identical DNA base-pairs.
You say there are five times more differences in genetic markers between humans and Neanderthals than amongst humans.
I'm only pointing out that the numbers of base-pair differences -- 3 million out of 3 billion total -- is the same between humans and Neanderthals as it is amongst various humans.
And I'm not disputing your time-line of divergences at all.
Clearly Neanderthals were more distantly related than our more recent out-of-Africa ancestors.
The issue here is: how "human" were Neanderthals?
Answer: plenty human -- human enough to:
So even if Neanderthals were not our human "brothers," they still were genetic cousins, and, so it appears, kissing cousins at that. ;-)