I take double-negative lessons from the late William F. Buckley, unfortunately.
Now, from what I’m reading - and these are people who actually live in Great Britain (England, specifically) - without EU protection, they fear refugees will be permitted to gain access to GB. Apparently, it is the EU who is holding them back in those Calais refugee camps. Calais has always been an entry point into England. It was once English territory.
OK, I dug a bit more. Apparently UK is paying for the fences and checkpoints in France, and does passport control on that side of the Channel, so if the agreement with France is ended, the British border controls would be at the English ferry terminals, not the French terminals. So the Brits probably would end up having to build concentration camps.