Not really - Canada and Australia have nearly as many already. The idea that the UK really has a lot of Muslims gets a bit exaggerated at times - some parts of the UK do, including some parts of London, but they really are quite a small minority overall.
But critically any free movement deal would likely only apply to citizens not just residents - and that makes a big difference. While some of these people do become citizens, a great many don't. That's part of the problem actually, in my view - that they don't feel connected enough to where they live now to make that commitment.