About 40% of colonials participated or were loyal to the Colonies. About 20% were Tories. The remaining 40% were fence sitters, waiting to see which way the wind blew.
I can't recall where I read this, but it referred to the number of colonists acting against the British before the "national" military effort in 1775... many were outraged, but only a handful were involved e.g. the Sons of Liberty, Stamp Act protests, Boston Tea Party, etc. Of course, when it was clear that the Colonial government would unite and support the effort by organizing a government sanctioned Army there was more support... but as you point out still not a majority.