Absolutely concur. And it just might be that simple.
Also my point (not to you) earlier about growth rate might apply. Who wants a 150 pound two year old in their house, who is trying to breed with every woman that walks by?
Yeah, it's definitely worth noting that Neanderthal females were burly critters too. In a one on one situation I think the Cro-Magnon guy would have to worry about getting raped more than the Neanderthal female.
My sister had four of those, not counting her husband.