Interesting point. I remember reading about 1/3 of the colonists were Loyalist. Most of them them left after the Revolution and they didn't all go back to England.
Indeed the first Canadians, apart from the French from what's called Lower Canada later (now Quebec), are the Loyalist Americans who remained loyal to the Crown in 1776. If I remember correctly, after 1783 most moved north to British North America and only a few returned to Britain.
Even today, their descendents still yield tremendous influence in political and cultural influence, particularly in Ontario. If you visit places like Kingston, Peterborough, Waterloo, Toronto, London, they have lots of monuments to the United Empire Loyalists. This is why Ontario on one hand is fiercely socialist (it is un-American), and on the otehr hand has the most sentimental attachment to the monarchy as the head of state of Canada (it is once again un-American to have a monarch as your HOS).