This doesn’t necessarily follow Brexit. The UK border controls in Calais have nothing to do with the EU - they’re a bilateral arrangement between Britain and France. All UK-bound travellers have to pass through them, all EU citizens included. This is because the UK has never been part of the Schengen open-border zone. So none of this would need to change after Brexit, unless one of the two parties decides that it should. (In the case of France not,of course, improbable.)
Then why are we reading - from people as well informed as you - that Britain, without the EU, will be overrun with the Calais refugees? I’m not taking a stand in this, it’s none of my business, but that is what is being said by people I know who live in GB. Admittedly, most of these folks are against Brexit.