Ben Franklin's eldest son, a noted Tory, had no problem "moving back home" to England.
Folks in the British North American colonies were quite free to relocate to anywhere in the empire (unless they were black slaves of course).
My family (on my maternal grandfather’s side) is rife with Benedict Arnold stories, as they were United Empire Loyalists who moved to Canada after the Revolution. There is said to be a non-direct family connection to the man.
Colonials who moved to England were always considered colonials, their loyalty to the Crown notwithstanding.