I would agree with that. However, I don't think the founding fathers ever anticipated that the electorate would hold their right (and responsibility) to vote in such low regard. The active decision to defend the country might be an indicator of an understanding of one's right and responsibility. Or not, as evidenced by Mr. Kerry.
While true, it's also somewhat irrelevant. Under the Constitution, as written by Madison, the States, not the federal government, determined who could vote, in Presidential elections or any other. Many states restricted voting to those who owned property. All restricted it to men.
For centuries, Switzerland required all men to show up with their militia weapon, or they could not vote. Worked pretty well for them.
In Presidential elections the state legislatures could directly appoint the electors if they so chose, or delegate the selection to the governor or a "blue ribbon commission", by passing the people altogether, except in so far as the actions of electors so chosen might affect their election chances at the next election.