Possibly the immigrants arrived during the Medieval warm period and did not do well when the climate turned colder in the little ice age.
Could be the women refused to be brood mares for their owners and had “post natal abortions” and/or killed or were killed because of this.
The strongest survived and the weakest parishes. Reinforcing the Viking DNA and the genetic strength to the climate, region (think less sunlight) and the ability to adapt. The mental and disease resistance reinforcement was also critical.