re: your point that Madison was just one of many with his own opinion...an honest reading of the Federalist papers, not to mention the quote offered from Jefferson, shows that Madison's view was indeed the majority position.
That's pure rubbish.
The Federalist Papers weren't even *written* by a majority of the Founders, and they certainly weren't ever voted into any legislation (a process that involves careful political compromises and a majority vote), so there is NO LEGITIMATE WAY that you can claim that such non-statute sources represent actual law (or even majority opinion, for that matter).