The Romans were stretched about as far as their empire had ranged during their four centuries of occupation, so yeah, not worth the trouble. As I read somewhere, the whole invasion was largely due to Claudius, emperor at that time, wanting to take Romans’ minds off of domestic issues. Some things just don’t change. And 400 years of Latin-speakers inhabiting the British Isles is why we have so much latin embedded in English, which is Germanic in origin. History is a fascinating thing; I wish I had more time to study it.
Claudius didn't have the physique to lead troops in battle, but he wanted to have at least one conquest in his legacy. In recent years the amount of political alliance, trade, and Roman patronage in Britain has been talked about, also the possibility that Gen. Agricola made a recon in force in Ireland. There's some evidence (not enough) that the Romans had a trading post and port just north of what is now Dublin -- not a conquest, just a presence. Claudius' successor Nero added to Roman holdings in Britain, and future emperor Vespasian rocketed across the entire south, in one season, reducing all the hillforts.
Actually old english took *very* few loanwords from provincial latin nor from p-celtic - the germans mainly displaced and enslaved romano-british except in the areas they did not conquer. the only notable celtic words to get into old english were iirc some geographic features (crag? hard to remember now). many romano-celtic (or earlier) place-names were replaced as well, so complete was the language replacement.
As far as I can tell p-celtic (welsh cornish bretonish) may have taken some, pont is one that comes to mind (bridge) but I never researched the issue in detail. aur (gold) maybe too but that is also in germanic (Ore) as the word itself is ancient, predating the languages in question by far enough to have a common root in IE.
where old english started getting a bunch of words of latin origin is the norman conquest, some through church latin but a lot more from old french.