But yes--you are right to imply that a good percentage of Loyalists came from NYC. The patriots fled when the brits occupied the city, many of them going to Philadelphia.
I once read that there were still many of dutch ancestory in NYC and they thought the British would win so they backed the Loyalists.