We do not find mixed communities. That means, we do not find communities of Neanderthal women 'taken' into human tribes. Now, if you want to just conjecture that Neanderthal women were abducted and taken into human tribes as some kind of work horse sex slaves, even though we find no archaeological evidence of that, then that's fine by me, but don't criticize me for making assumptions!
My point was that human social norms would dictate that you would find this, but we don't, so what gives? To me the lack of this evidence shows that it didn't happen. The fact that it does happen with human bands, but not in this case is indicative that there was something stronger than normal social exclusion at work.