Contrary to popular Libertarian sentiment, the American Republic was never a dictatorship run by Madison. Instead, Madison was merely *one* minority vote among numerous Founding Fathers.
Moreover, the "alternative" is not between "everything" or "nothing", but rather somewhat less stark than that, as countless court cases over the last 2 centuries reveal.
re: your point that Madison was just one of many with his own opinion...an honest reading of the Federalist papers, not to mention the quote offered from Jefferson, shows that Madison's view was indeed the majority position.