You're right, it's largely a labour market issue. In the medium term, contrary to the general assumption, it's the skilled rather than the unskilled labour market in Britain which will be most affected. Take, for example, the position of almost any British plumbing contractor at the moment. Does he take on an unskilled teenage apprentice, and train him up for four or five years before he's qualified to work unsupervised and the employer begins to get a return on his investment? Or does he buy "off the shelf", and at far lower cost, a highly skilled, fully qualified and highly motivated Polish plumber? Extrapolate this to all skilled trades over a decade or so, and it's not hard to see the pattern which would emerge.
Honestly, while it will be painful for the short term, in the long term, this is a good thing, helping to squeeze inefficiencies out of the labor market. It just sucks for the guys stuck in the meantime. Adapt or die, creative destruction, in other words, capitalism at work.