At the time of the Germanic Invasion of Britannia, the inhabitants were called Britons, as in Arthur, King of the Britons, for you Monty Python fans. These people were pushed to the west -- to Wales. Many others settled in France -- Brittany. The Angles/Saxons/Jutes did not kill everyone, they just forced their capitulation. Eire was spared this invasion. Subsequently, England was invaded by Danish Norsemen, and Eire by Norwegian Norsemen, (the Fitzes). Each wave of invaders imposed their language and culture on the conquered peoples. The Irish, Scots, and Welsh all speak a form of the language of the Celts -- Gaelic or Gallic, as in Gaul.
When the Anglo Saxons invaded, some of the Celts were pushed west. But many of them remained in place as serfs and eventually intermarried with the conquerers or had illegitimate children with them to much the same effect, putting some Anglo Saxon blood among the peasants and sometimes some Celtic blood among the nobles.
As you suggest, the earliest Arthurian stories are Celtic.
Mostly it was just that the mountains of Wales and Scotland make them harder places to conquer, so the native Celts were able to maintain their rule in those places.