He was considered a person of slight character. But not an "immigrant" in the sense of a non-British subject. He was NOT considered "English" though.
Ben Franklin's eldest son, a noted Tory, had no problem "moving back home" to England.
Folks in the British North American colonies were quite free to relocate to anywhere in the empire (unless they were black slaves of course).