Not just "some", most. As you've pointed out, eligible voters were a minority, even among free white males. But all free white males, of the proper ages, were required to be in the militia, or more properly, they *were* the militia.
It's not an exception at all.
I did point out that eligible voters were a minority of the population, yes. Less than 20%. But I never said eligible voters were a minority among free white males.
If you wish to make that point, feel free to do so. But I would like some references.