The Loyalists were originally from NY State, and more than likely moved to Canada when the war broke out. I often wonder if the young man had fallen in love with the Loyalist daughter while they were still in NY, then gave it all up after the war, to go to Canada and marry the girl of his dreams. Nice story, but no proof. Both families were from Duchess County, NY, so it's definitely a possibility.
You might want to check to see if your Loyalist ancestors filed any claims against the US government after the Revolutionary War. Many Loyalists lost property, goods, homes, etc., when they left. There was a method set up after the War to file claims for restitution. Never can tell what you may find but any claims could show exactly where your ancestors lived, worked, etc.