The Romans never invaded Scotland, because Scotland didn't exist until the 7th century; the ancestors of the Scots were still in Ireland.
The reason Hadrian built his wall is the same as why did he pull out of Mesopotamia after Trajan's troops had washed their weapons in the Persian Gulf, and why he almost gave up gold-rich Dacia -- because he wanted to spend his time banging young boys in the ass instead of leading the army. It was also said of Hadrian that he knew every soldier in the army by name -- a ridiculous notion, but it's easy to see how it could have arisen.
Agricola was ready to wrap up Caledonia, but was recalled. The problem was always that there was little of value in the land to be conquered, and pressing needs elsewhere in the Empire.
Another wall (Antonine’s Wall) was built subsequently about 70 miles to the north of Hadrian’s wall. There was an interesting article on that in a recent Archaeology magazine, I forget which one, will look for it and post info if requested. Marcus Aurelius decided that the British enterprise was not worth maintaining and pulled the Romans out.