There were still lots of Tories who had stayed instead of populating Canada.
...and the Southern States were worried of slave rebellions.
bttt