Any polity that gave 'em too much crap got their asses kicked. :^) The Romans only operated in force in what is now Scotland four times, beginning with Agricola, and ending with Septimius Severus. Hadrian's Wall (as well as the other "limes" and wall systems such as the one in Germany and the much less well known one is the Middle East) offered a sturdy defensive barrier, but mainly served the role of the great hedge in British India, controlling access and trade and excise tax flow.
And every time they got kilt..............