Bingo! That is the take-away.
Moreover, scholars claim that group of Patriots was less than the 80% of the American population.
That is to say 80% wished to sit out the conflict or were aligned in some way with the most powerful army.
Ergo, they may be able to capture the popular vote or thwart the Electoral College but there just isn't enough of them.
3 million military,2 million police, etc......... 5 million
American gun owners.....100 million