The "people at large" was a phrase used to refer to a select group of individuals -- not all. In this instance, it's a reference to a well regulated state Militia.
In Federalist #29, Hamilton is actually arguing against having all individuals armed and trained to fight, saying that doing so "would be a real grievance to the people, and a serious public inconvenience and loss."